Stack target
| Technique | Trigger | Win | Plan |
|---|---|---|---|
| Parakeet and NeMo family routing | Compatible ASR family | Better family-specific defaults | All plans |
| VAD chunk batching | Long-form audio | Higher throughput on chunked audio | Core |
| INT8 encoder (CTranslate2) | Compatible GPU and runtime | Lower memory and higher throughput | Core |
| Greedy vs beam decode tuning | Throughput vs accuracy target | Faster decode at beam=1, accuracy at beam=5 | Core |
ASR is non-autoregressive at the sequence level, so speculative decoding does not apply. There is no Distil-Whisper draft path or cross-chunk encoder cache in the candidate generator today. The sweep tunes compute type, beam size, and VAD chunk batching only.
Quality gates
| Metric | Ceiling | Source |
|---|---|---|
| WER absolute delta | Under 1.5 percentage points | Holdout transcription set |
| Real-time factor | Under 0.3 | Faster than real-time when under 1.0 |
| Quality floor | At least 0.85 | RunInfra ranking gate |
asr_transcription JSONL test set when each scored row includes audioBase64 and reference, and the audio decodes to WAV with a parseable duration. Rows may also include contentType or mimeType; data URL prefixes are stripped before benchmarking. The backend caps inline eval at 16 labeled samples, rejects compressed or unknown-duration audio before paid custom WER benchmarking, and promotes only when the measured corpus WER gate passes.
Detection, routing, and application
| Decision | Selection rule | Runtime behavior |
|---|---|---|
| Compute type | GPU family and model support | Selects fp16 or int8 where safe |
| Beam size | Accuracy and latency target | Sets decoder beam size for transcription |
| VAD filter | Long-form or noisy audio | Splits speech regions before decoding |
| Batched chunks | Long files with many speech regions | Processes compatible chunks together |
| ASR family | Model architecture and config | Chooses family-specific defaults for Whisper, NeMo, Parakeet, and related models |
Verification
- The plan should show compute type, beam size, and VAD choices.
- The benchmark response should include the effective ASR runtime settings.
- Long-form tests should report both throughput and WER against the quality gate.
Product validation surfaces
ASR optimization evidence comes from the Modal optimization session, not from a single warm preview request alone. For CTranslate2 and faster-whisper candidates, compare baseline and optimized rows by WER, real-time factor, audio seconds per second, cost per request, and the selected runtime settings. A valid promotion keeps WER inside the quality gate while improving throughput or real-time factor. The dashboard Code tab should receive the generated ASR serving artifact after a measured optimization version is persisted. For faster-whisper and CTranslate2 winners, the generatedserve.sh starts a FastAPI server for /v1/audio/transcriptions, exposes /health, and carries the selected COMPUTE_TYPE, BEAM_SIZE, BATCH_SIZE, TASK, VAD_FILTER, and timestamp settings. If the Code tab shows only a generic LLM or vLLM script for an ASR CTranslate2 winner, treat it as a code-generation sync bug.
The dashboard Test tab can compare the baseline and optimized ASR paths through the dual-run endpoint. Uploaded audio may arrive from browsers or object storage as application/octet-stream or binary/octet-stream; the Engine normalizes those generic ASR uploads to audio/wav before sending them to the vLLM-Omni transcription worker. vLLM-Omni ASR can take longer on the first cold baseline load, so the poll budget is longer than the faster-whisper path.
Do not judge ASR optimization only from one warm single-clip latency number. A short clip can show similar baseline and optimized latency after both runtimes are warm. The expected ASR win for CTranslate2 and faster-whisper is higher audio throughput, lower real-time factor, lower cost per request, or lower memory at the same measured WER.
Rollback
Availability notes
- VAD chunk batching helps most on long-form audio with many speech regions. The faster-whisper batched pipeline parallelizes chunks within a single audio, not across multiple audios.
- INT8 encoder via CTranslate2 lowers memory and raises throughput, with a small WER drift that the quality floor catches. A/B per language is recommended for accented or code-switched audio.
- Greedy decode (beam=1) trades a small accuracy hit for faster throughput. Beam=5 stays the accuracy reference.
- NeMo and Parakeet routing depends on runtime support for the selected checkpoint.