> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# ASR optimization runbook

> Speech-to-text optimization techniques, quality gates, verification paths, and rollback controls.

## Stack target

| Technique                        | Trigger                       | Win                                         | Plan      |
| -------------------------------- | ----------------------------- | ------------------------------------------- | --------- |
| Parakeet and NeMo family routing | Compatible ASR family         | Better family-specific defaults             | All plans |
| VAD chunk batching               | Long-form audio               | Higher throughput on chunked audio          | Core      |
| INT8 encoder (CTranslate2)       | Compatible GPU and runtime    | Lower memory and higher throughput          | Core      |
| Greedy vs beam decode tuning     | Throughput vs accuracy target | Faster decode at beam=1, accuracy at beam=5 | Core      |

<Callout type="info">
  ASR is non-autoregressive at the sequence level, so speculative decoding does not apply. There is no Distil-Whisper draft path or cross-chunk encoder cache in the candidate generator today. The sweep tunes compute type, beam size, and VAD chunk batching only.
</Callout>

## Quality gates

| Metric             | Ceiling                     | Source                               |
| ------------------ | --------------------------- | ------------------------------------ |
| WER absolute delta | Under 1.5 percentage points | Holdout transcription set            |
| Real-time factor   | Under 0.3                   | Faster than real-time when under 1.0 |
| Quality floor      | At least 0.85               | RunInfra ranking gate                |

ASR live lab execution can use a user-uploaded `asr_transcription` JSONL test set when each scored row includes `audioBase64` and `reference`, and the audio decodes to WAV with a parseable duration. Rows may also include `contentType` or `mimeType`; data URL prefixes are stripped before benchmarking. The backend caps inline eval at 16 labeled samples, rejects compressed or unknown-duration audio before paid custom WER benchmarking, and promotes only when the measured corpus WER gate passes.

## Detection, routing, and application

| Decision       | Selection rule                      | Runtime behavior                                                                 |
| -------------- | ----------------------------------- | -------------------------------------------------------------------------------- |
| Compute type   | GPU family and model support        | Selects fp16 or int8 where safe                                                  |
| Beam size      | Accuracy and latency target         | Sets decoder beam size for transcription                                         |
| VAD filter     | Long-form or noisy audio            | Splits speech regions before decoding                                            |
| Batched chunks | Long files with many speech regions | Processes compatible chunks together                                             |
| ASR family     | Model architecture and config       | Chooses family-specific defaults for Whisper, NeMo, Parakeet, and related models |

## Verification

1. The plan should show compute type, beam size, and VAD choices.
2. The benchmark response should include the effective ASR runtime settings.
3. Long-form tests should report both throughput and WER against the quality gate.

## Product validation surfaces

ASR optimization evidence comes from the Modal optimization session, not from a single warm preview request alone. For CTranslate2 and faster-whisper candidates, compare baseline and optimized rows by WER, real-time factor, audio seconds per second, cost per request, and the selected runtime settings. A valid promotion keeps WER inside the quality gate while improving throughput or real-time factor.

The dashboard **Code** tab should receive the generated ASR serving artifact after a measured optimization version is persisted. For faster-whisper and CTranslate2 winners, the generated `serve.sh` starts a FastAPI server for `/v1/audio/transcriptions`, exposes `/health`, and carries the selected `COMPUTE_TYPE`, `BEAM_SIZE`, `BATCH_SIZE`, `TASK`, `VAD_FILTER`, and timestamp settings. If the Code tab shows only a generic LLM or vLLM script for an ASR CTranslate2 winner, treat it as a code-generation sync bug.

The dashboard **Test** tab can compare the baseline and optimized ASR paths through the dual-run endpoint. Uploaded audio may arrive from browsers or object storage as `application/octet-stream` or `binary/octet-stream`; the Engine normalizes those generic ASR uploads to `audio/wav` before sending them to the vLLM-Omni transcription worker. vLLM-Omni ASR can take longer on the first cold baseline load, so the poll budget is longer than the faster-whisper path.

Do not judge ASR optimization only from one warm single-clip latency number. A short clip can show similar baseline and optimized latency after both runtimes are warm. The expected ASR win for CTranslate2 and faster-whisper is higher audio throughput, lower real-time factor, lower cost per request, or lower memory at the same measured WER.

## Rollback

```ts theme={"dark"}
constraints.customerOverrides = {
  skipTechniques: ["quant"],
};
```

This disables INT8 encoder variants and keeps fp16 candidates.

## Availability notes

* VAD chunk batching helps most on long-form audio with many speech regions. The faster-whisper batched pipeline parallelizes chunks within a single audio, not across multiple audios.
* INT8 encoder via CTranslate2 lowers memory and raises throughput, with a small WER drift that the quality floor catches. A/B per language is recommended for accented or code-switched audio.
* Greedy decode (beam=1) trades a small accuracy hit for faster throughput. Beam=5 stays the accuracy reference.
* NeMo and Parakeet routing depends on runtime support for the selected checkpoint.