Safe measurement plan · Functional compatibility

Measure tools the way Warp sends them

Use 33 inert schemas and separate native tool use from sidecar adaptation.

Primary local sources: plan, model matrix, per-cell plan, manifest and sanitized inventory.

The idea in plain language

The test runs no shell and no network. It exposes 33 inert tools, requests one deterministic function and returns the result locally. Success requires a name, valid JSON and correct use of the result — not merely HTTP 200.

Think of this measurement as a sealed scale: if you swap the scale or move the object midway, the number loses meaning. The analogy breaks because gateways can also transform the request, not merely weigh it.

The sidecar injects a fish hint when tools exist. Qwen, DeepSeek and GLM get tool_choice normalized to auto. Kimi tool history is flattened into text. Those results are emulated compatibility where applicable, not native capability.

Verifiable excerpt

tools = 33 inert schemas\nfunction = get_benchmark_nonce\nside_effects = none

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/benchmark-cell-plan.csv · outputs/planned-run-manifest.json

The path in one picture

Warp → sidecar → CLIProxyAPI → provider → proof gate

Retrieval check

Is HTTP 200 enough to approve tools?