Safe measurement plan · The observable stream

Define metrics that keep their meaning

Use monotonic milestones for TTFT, TTFE, visible TPS and total latency.

Primary local sources: plan, model matrix, per-cell plan, manifest and sanitized inventory.

The idea in plain language

The clock starts when the final request byte is sent. The first event is not necessarily the first token you can see. The plan therefore measures TTFE and TTFT separately. TPS uses visible tokens only, between the first and last visible deltas.

Think of this measurement as a sealed scale: if you swap the scale or move the object midway, the number loses meaning. The analogy breaks because gateways can also transform the request, not merely weigh it.

TTFT = t_first_visible − t_send. TTFE = t_first_event − t_send. Visible TPS = visible tokens ÷ (t_last_visible − t_first_visible). Total latency = t_done − t_send. Reasoning before visible text gets a separate metric.

Verifiable excerpt

TTFT = t_first_visible - t_send\nTPS  = visible_tokens / visible_seconds\nTOTAL = t_done - t_send

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/benchmark-cell-plan.csv · outputs/planned-run-manifest.json

The path in one picture

Warp → sidecar → CLIProxyAPI → provider → proof gate

Retrieval check

Which event ends the headline TTFT?