Skip to main content
A run is only worth paying for if it can tell you what changed. RunInfra ends every run with one of two things: a before and after you can act on, or a statement of exactly which piece of evidence is missing. There is no third option where a number appears without the evidence behind it.

​
Two measurements, one instrument

A speedup is a ratio, and a ratio is only meaningful when both sides were produced the same way. Two latency numbers taken from different harnesses are not a before and after, they are two unrelated facts. Dividing one by the other publishes the gap between the instruments and calls it your improvement. So every measured result RunInfra records now carries four facts about itself, alongside the number: Two rows are compared only when all four agree. This is why a run can measure real work and still decline to publish a ratio for it.

​
When a comparison is refused

A refusal is a correct outcome, not an error. When the evidence does not support a fair comparison, RunInfra crowns no winner and states which axis differed and what the two values were, so you can tell at a glance whether it is a real limitation or something you can fix by re-running.
A refused comparison does not fail the run. The measurements it produced stay visible and stay yours. What is withheld is only the ratio that the evidence does not support.

​
The baseline is measured, not assumed

The baseline a run compares against is measured during that run, on the same instrument as the optimized results, before any technique is applied. It is never synthesized from a catalog figure, an earlier run, or a learned prior. If the baseline measurement did not run, the run has no baseline and says so, rather than substituting a number that would make the ratio look reasonable. A borrowed baseline is the easiest way to manufacture a speedup, and it is the one shortcut this contract exists to close.

​
Every run carries a quality verdict

Speed without an accuracy answer is half a result. Every completed run now records exactly one quality verdict, and there are three possible values. A missing verdict is never acceptable. Not measured is an honest answer and it is written down as one. Silence is not, because a run that skipped the gate would otherwise be indistinguishable from a run that passed it. Two details are worth knowing:
  • Quantized KV cache counts as output-affecting, even though the weights are untouched, because it changes attention arithmetic.
  • Speculative decoding does not count, because it is output-identical when its equivalence check passed, and an unverified one is never promoted.
When several stages in one run carry accuracy gates, the lowest score decides the verdict. A pipeline is only as accurate as its weakest stage, and reporting the best one would let a passing stage hide a failing one.

​
What this means for reading a result

  • A published speedup means both sides were measured in this run, on one instrument, on one GPU, for one model. Read the quality verdict beside it before treating the speed number as the whole result.
  • A named refusal means the work happened but the evidence does not support a ratio. Read the named axis to see what to change.
  • A not-measured quality verdict means treat the speed number as unaccompanied. It is not a failure, and it is not a pass.
  • Quality gates for how an accuracy check decides promotion, and why a check that could not run is never a pass.
  • Optimization for how candidates are searched, ranked, and settled.
  • Lossless first for why a run may deliberately apply nothing that could affect quality.
  • Benchmarks methodology for how RunInfra reports measurements.