Run outcomes

A run ends with one of two things: a before and after you can act on, or a statement of exactly which piece of evidence is missing. There is no third option where a number appears without the evidence behind it.

​

Reading your result

• A published speedup means both sides were measured in this run, on one instrument, on one GPU, for one model. Read the quality verdict beside it before you treat the speed number as the whole result.
• A named refusal means the work happened but the evidence does not support a ratio. Read the named axis to see what to change.
• A not-measured quality verdict means you should treat the speed number as unaccompanied. It is not a failure, and it is not a pass.

​

When a comparison is refused

A ratio is only meaningful when both sides were produced the same way. Two latency numbers from different harnesses are not a before and after, they are two unrelated facts, and dividing one by the other publishes the gap between the instruments and calls it your improvement. So every measured result carries four facts about itself alongside the number: whether it is the baseline or an optimized result, which instrument produced its latency, which unit its throughput is in, and which model and modality it belongs to. Two rows are compared only when all of them agree, which is why a run can measure real work and still decline to publish a ratio for it.

​

The baseline is measured, not assumed

The baseline a run compares against is measured during that run, on the same instrument as the optimized results, before any technique is applied. It is never synthesized from a catalog figure, an earlier run, or a learned prior. If the baseline measurement did not run, the run has no baseline and says so, rather than substituting a number that would make the ratio look reasonable. A borrowed baseline is the easiest way to manufacture a speedup, and it is the one shortcut this contract exists to close.

​

Every run carries a quality verdict

Speed without an accuracy answer is half a result. Every completed run records exactly one quality verdict, and there are three possible values. Two details are worth knowing:

• Quantized KV cache counts as output-affecting, even though the weights are untouched, because it changes attention arithmetic.
• Speculative decoding does not count, because it is output-identical when its equivalence check passed, and an unverified one is never promoted.

When several stages in one run carry accuracy gates, the lowest score decides the verdict. A pipeline is only as accurate as its weakest stage, and reporting the best one would let a passing stage hide a failing one.

​

Related