Skip to main content
Speed is easy to measure and easy to sell. The harder question is whether the faster candidate is still the model you asked for. Before RunInfra promotes an optimized candidate over your baseline, that candidate has to clear a quality gate for its modality.

​
Three outcomes, not two

A quality check has three possible outcomes, and the third one is the point of this page. Not evaluated is not a pass and it is not a fail. It is its own state, it is labelled as its own state, and it never quietly becomes either of the other two. This is a product promise, not an implementation detail: RunInfra would rather tell you a number is missing than invent one. A check resolves to not evaluated when there is genuinely nothing to report, for example when no runner is available for that benchmark, when a runner does not return a usable number, or when the metric’s scale cannot be judged. In every one of those cases the candidate is held back rather than promoted.
A not-evaluated result is the reason a run can complete successfully and still decline to promote a winner. The run did the work; the evidence needed to certify the result was not there. Treat it as “we do not know yet”, never as “it is fine”.

​
What each modality is measured on

Each modality has canonical benchmarks that define acceptance for it. These are the authoritative pre-promotion checks, distinct from the cheap in-loop signals a sweep emits while it searches. If a modality has no gate benchmark at all, RunInfra profiles and reports it but never promotes a tuned candidate on quality grounds. Reporting without promoting is the honest option when there is no acceptance test to pass.

​
Both axes have to agree

Promotion needs a measured performance win and a quality result within budget. Either axis can block it on its own:
  • A candidate whose quality is not evaluated is never promoted, however fast it measured.
  • A candidate whose speed difference sits inside run-to-run noise is not promoted either, and no speedup is claimed for it.
When a comparison cannot be made coherently, the verdict itself becomes unmeasured and no ratio is published. Withholding the verdict, rather than publishing a number with a caveat attached, is deliberate.

​
Text to speech and certified runtimes

Speech synthesis carries an extra gate. RunInfra maintains a list of model families whose runtime behaviour has been certified, and a family outside that list is not silently run as though it had been.
  • A model family with no certification entry is gated with an explicit reason saying the compatibility check did not run for that family. The phase is gated and costs you nothing.
  • A model family that has an entry but does not match on runtime, task, or GPU is refused for that specific mismatch.
Profiling an uncertified family can still proceed, because blocking measurement would be self-defeating: with no measurement there is no evidence, and with no evidence the family could never earn a certification entry. Those profiling results carry the not-evaluated marker so nothing downstream can read a measurement as a certification. The distinction is worth stating plainly. An evaluated refusal, such as a malformed request or a runtime check that ran and failed, blocks everything. Only the absence of a check is treated differently depending on whether you are measuring or promoting.