> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Quality gates

> How RunInfra decides whether an optimized candidate is good enough to promote, and why a check that could not run is reported as not evaluated rather than as a pass.

Speed is easy to measure and easy to sell. The harder question is whether the faster candidate is still the model you asked for. Before RunInfra promotes an optimized candidate over your baseline, that candidate has to clear a quality gate for its modality.

## Three outcomes, not two

A quality check has three possible outcomes, and the third one is the point of this page.

| Outcome           | What it means                                                                                            | Is the candidate promoted?          |
| ----------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| **Passed**        | The check ran, and the measured result stayed within the accuracy budget and above the modality's floor. | Yes, if it also won on performance. |
| **Failed**        | The check ran, and the measured result fell outside the budget or below the floor.                       | No.                                 |
| **Not evaluated** | The check could not run at all, so there is no measured result.                                          | No.                                 |

**Not evaluated is not a pass and it is not a fail.** It is its own state, it is labelled as its own state, and it never quietly becomes either of the other two. This is a product promise, not an implementation detail: RunInfra would rather tell you a number is missing than invent one.

A check resolves to not evaluated when there is genuinely nothing to report, for example when no runner is available for that benchmark, when a runner does not return a usable number, or when the metric's scale cannot be judged. In every one of those cases the candidate is held back rather than promoted.

<Warning>
  A not-evaluated result is the reason a run can complete successfully and still decline to promote a winner. The run did the work; the evidence needed to certify the result was not there. Treat it as "we do not know yet", never as "it is fine".
</Warning>

## What each modality is measured on

Each modality has canonical benchmarks that define acceptance for it. These are the authoritative pre-promotion checks, distinct from the cheap in-loop signals a sweep emits while it searches.

| Modality           | Gate benchmarks   |
| ------------------ | ----------------- |
| LLM                | MMLU, HumanEval   |
| Embedding          | MTEB              |
| Rerank             | NDCG\@10, MRR\@10 |
| Speech recognition | WER               |
| Text to speech     | MOS, WER          |
| Voice pipeline     | WER, MOS          |
| Image generation   | FID, CLIP         |
| Video generation   | CLIP              |
| Vision language    | VQA, MMMU         |
| Translation        | BLEU, COMET       |

If a modality has no gate benchmark at all, RunInfra profiles and reports it but never promotes a tuned candidate on quality grounds. Reporting without promoting is the honest option when there is no acceptance test to pass.

## Both axes have to agree

Promotion needs a measured performance win **and** a quality result within budget. Either axis can block it on its own:

* A candidate whose quality is not evaluated is never promoted, however fast it measured.
* A candidate whose speed difference sits inside run-to-run noise is not promoted either, and no speedup is claimed for it.

When a comparison cannot be made coherently, the verdict itself becomes unmeasured and no ratio is published. Withholding the verdict, rather than publishing a number with a caveat attached, is deliberate.

## Text to speech and certified runtimes

Speech synthesis carries an extra gate. RunInfra maintains a list of model families whose runtime behaviour has been certified, and a family outside that list is not silently run as though it had been.

* A model family with **no certification entry** is gated with an explicit reason saying the compatibility check did not run for that family. The phase is gated and costs you nothing.
* A model family that **has an entry but does not match** on runtime, task, or GPU is refused for that specific mismatch.

Profiling an uncertified family can still proceed, because blocking measurement would be self-defeating: with no measurement there is no evidence, and with no evidence the family could never earn a certification entry. Those profiling results carry the not-evaluated marker so nothing downstream can read a measurement as a certification.

The distinction is worth stating plainly. An **evaluated refusal**, such as a malformed request or a runtime check that ran and failed, blocks everything. Only the **absence** of a check is treated differently depending on whether you are measuring or promoting.

## Related

* [Optimization](/docs/features/optimization) for how candidates are searched and ranked.
* [Benchmark reports](/docs/features/benchmark-reports) for running named benchmarks against a promoted winner.
* [Text to speech runbook](/docs/runbooks/tts) for the TTS optimization path.
* [Benchmarks methodology](/docs/reference/benchmarks-methodology) for how RunInfra reports measurements.