Safe measurement plan · The observable stream

Make the stream tell the truth

Separate headers, valid SSE, reasoning, visible content and the terminal event.

Primary local sources: plan, model matrix, per-cell plan, manifest and sanitized inventory.

The idea in plain language

A stream may open quickly and stay silent before showing text. It may also close without a terminal event. The harness keeps every milestone and calls the sample successful only when the flow ends coherently.

Think of this measurement as a sealed scale: if you swap the scale or move the object midway, the number loses meaning. The analogy breaks because gateways can also transform the request, not merely weigh it.

The clock is monotonic and high resolution. Tokens come first from terminal usage separated from reasoning; then an exact tokenizer; finally a labelled estimate. Without an honest basis, TPS is N/A and characters/s remains auxiliary.

Verifiable excerpt

t_send → t_headers → t_first_event\n       → t_first_reasoning → t_first_visible\n       → t_last_visible → t_done

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/benchmark-cell-plan.csv · outputs/planned-run-manifest.json

The path in one picture

Warp → sidecar → CLIProxyAPI → provider → proof gate

Retrieval check

Without reliable tokenization, what happens to TPS?