A number you cannot trace is not a number

The most useful thing this project built was not a kernel or a config. It was a measurement discipline strict enough that it kept deleting our own results. Four published claims were withdrawn by the people who made them, each one against our own interest, and the reason the surviving numbers are worth anything is that those four are on the record beside them.

This is how the instrument works and what it cost us.

The window: one container, one wheel, one variable

Every publishable comparison in this repository comes from a two-column window. Both arms run sequentially inside ONE container, on THE SAME eight B300 cards, with ONE wheel install, so the comparison carries no node-to-node variance and no install-to-install variance. Between arms the harness drains VRAM and records what it found: in the bring-up window, a maximum of 726.0 MiB used across all eight GPUs against a 2048.0 MiB threshold, both times.

Before any number renders, the comparison tool compares the two arms field by field and refuses if they differ. In the headline window it verified 26 request-profile fields identical (results/raw/k4pub-comparison.json, profile_verified.verified true): 11 single-stream fields covering concurrency, requests, warmup requests, input and output token targets, temperature, top_p, streaming, ignore_eos and seed; and 15 sweep fields covering workload, input and output token counts, GPU count, seed, concurrency list, and per-point requests, warmup and prompt fingerprint. The fingerprints are full sha256 digests of the prompt set at each of concurrency 1, 8 and 32. Two arms that disagree about what they were asked are not a comparison, and the tool will not pretend otherwise.

The delta convention is fixed and printed in every output: positive always means arm B is better, throughput uses (B minus A) over the absolute value of A, latency uses (A minus B). Any value absent from a raw file renders as the word absent and its delta is absent too. No value is interpolated, derived from a neighbouring percentile, or invented.

The refusal codes

The tool exits with a named refusal rather than a best effort: 2 USAGE, 3 MISSING_ARM, 4 PROFILE_MISMATCH, 5 ZERO_BASELINE, 6 PROFILE_UNVERIFIABLE. Refusing on a profile it cannot check is a distinct code from refusing on a profile that does not match, because "I could not verify" and "I verified and it differs" are different facts.

This fired for real. In the first custom-prefill plus DSpark window, stock upstream failed to serve, so arm A produced no benchmark artifact at all. The comparison step exited 3 with REFUSED MISSING_ARM: arm A single-stream raw file ... cannot be read (results/raw/twocol-step-compare.log), and the payload recorded headline.publishable false, headline.armA null, headline.ratio_b_over_a null. There is no stock-versus-tuned number for that window and there never will be, which is inconvenient, because a serves-at-all result is the most quotable thing that window produced.

The noise floor, measured rather than assumed

Two servers on the same hardware, running the same stock configuration, were benched against each other as an A/A control (results/raw/serving-c1-c8-runA.json and results/raw/serving-c1-c8-runB.json). The deltas: minus 0.33 and plus 0.39 percent on tok/s/GPU, plus 0.28 and minus 0.00 percent on TPOT p50, minus 0.05 and plus 0.74 percent on TTFT p50, at concurrency 1 and 8. That establishes a harness noise floor of roughly plus or minus 0.7 percent TTFT, 0.4 percent throughput and 0.3 percent TPOT, under the stated conditions: workload A, 3 requests per point, max_tokens 256. Any claim below that floor is noise.

That floor is what lets a later result be read honestly rather than spun. When the shipped configuration measured minus 0.83 percent throughput at concurrency 32, the correct reading was "flat", not "a small regression" and certainly not "no change".

A second, larger spread is also on the record: the same configuration measured 118.27 tok/s single stream in one window and 119.84 in another, a 1.3 percent run-to-run spread between windows that is wider than the within-window throughput floor. Recorded rather than smoothed.

That A/A control also paid for itself twice over by finding two harness defects before they could contaminate a claim. The patch hook armed on the wrong module, so the candidate arm ran stock and the whole run was an A/A rather than an A/B. And the accuracy gate turned out to be unsound, which is the first of the four withdrawals.

Four things we withdrew

This matters most where it hurts. The W4A8 expert backend later failed that same gate at top-1 65.94 percent, KL 0.5364, with 24.16 percent of positions skipped for thin shared support (results/raw/w4a8gate-gate-result.json). Sitting below the 78.12 percent self-comparison control is weak evidence that the backend diverges more than instance noise, and that is expected by construction, because W4A8 quantizes activations and genuinely should differ. What the instrument cannot say is whether the difference costs task quality, which is the only question a buyer has. The tool reports task-level evaluation as SKIPPED and its own note names the runners as unwired. So the backend stays out of the product, and the number it produced stays a recorded fact about a configuration we do not sell.

The raw-evidence law, and what it cost

Every published number traces to a committed raw file with per-request rows. The rule has teeth because it deleted our best-looking table. The first healthy serve of this model produced aggregate throughput of 49.0, 191.0, 299.7 and 329.7 tok/s at concurrency 1, 8, 32 and 128. That serve revision wrote its raw JSON inside the container and the container died before persistence to the volume existed. Those four rows have no raw file, so they are uncitable forever. They stay in the log labelled as recorded history, and every published claim uses the lower, raw-backed points instead.

The same law caught a citation that would have looked like fabrication. An early draft cited engagement evidence as kda.py:163, which is the vLLM source location that emitted the message, not a line number in our log. A checker following it would have landed on a safetensors progress bar. The correction is in the log: the kernel engagement line is log line 133 of the committed serve log, and it is cited that way now.

Two windows were lost outright to a VRAM drain defect, at roughly 50 dollars an hour on eight B300s. The fix was to stop guessing from process names and ask the driver who holds the memory, and to record what the sweep killed either way, so the next occurrence is one read rather than another window.

The thesis

None of this makes a number bigger. It makes a number survivable. A reader with a terminal can open every raw file cited in these writeups, recompute the p50 from the eight per-request rows, check that the two arms were asked the same question, and find the places where we scored ourselves badly and said so. That is the only thing that makes the rest believable, and it is cheaper than being caught.