> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio-LM and voice-pipeline optimization runbook

> Optimization targets for audio-language models and co-located voice pipelines.

## What runs today, audio-language models

Audio-language models (Qwen2-Audio, Ultravox style) are kept out of the text-only benchmark path and swept with an audio-bearing probe (a synthetic WAV plus a short text instruction). The sweep tunes the same serving-config knobs we use for text LLMs, with conservative multimodal limits because audio encoder tokens make each prompt heavier than a text-only request.

| Technique                                | Trigger                     | Win                                                                                         |
| ---------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------- |
| Serving-config sweep with an audio probe | Audio-language model family | Finds a latency vs throughput configuration measured on real audio input                    |
| Conservative multimodal limits           | Audio-language prompts      | Caps context length, sequence count, and batched tokens so audio-heavy requests stay stable |

<Callout type="note">
  The audio-language path tunes serving configuration on a real audio probe. It does not change model weights or wire bespoke codec, cross-modal KV, or encoder-precision passes. Those are planned, see below.
</Callout>

## Voice pipeline status

<Callout type="warning">
  In development. The Modal schema exposes a native `/voice-pipeline-benchmark` endpoint and stale-deploy health capability, and the lab runner has an opt-in native adapter slot. Live voice-pipeline optimization remains blocked by default unless the suite manifest carries explicit ASR, LLM, TTS, audio, and reference-text stage config. The current native adapter accepts only a same-model, same-revision single-process co-resident path, for example one Omni model handling ASR, reasoning, and speech. Arbitrary ASR plus LLM plus TTS stacks still fail closed before paid Modal work until the multi-process co-resident worker lands. Do not treat the current endpoint as certified paid evidence for arbitrary three-model voice stacks.
</Callout>

| Technique                                | Current status                                                                                                                    | Intended win                                                       |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| Native voice-pipeline benchmark endpoint | Schema and route wired; same-process configs can emit owner-gated Modal lab commands; arbitrary multi-model stacks remain blocked | Measures ASR, LLM, TTS, and full turn latency through one contract |
| Streaming audio codec (Mimi style)       | Planned                                                                                                                           | Lower end-to-end latency                                           |
| Cross-modal KV management                | Planned                                                                                                                           | Better memory use across audio and text                            |
| Encoder precision tuning                 | Planned                                                                                                                           | Lower memory without changing LLM weights                          |
| Stage pipelining                         | Endpoint accepts the knobs, production evidence still gated                                                                       | Starts downstream work before the full utterance finishes          |
| Frame scheduling                         | Planned                                                                                                                           | More stable turn-taking latency                                    |
| Co-located media and inference           | Requires follow-up multi-process co-resident worker for arbitrary three-model stacks                                              | Avoids cross-service hops                                          |

## Planned latency budget, voice pipeline

These targets describe the planned voice-pipeline path, not current certified production behavior.

| Component           | Target                         | Notes                                                  |
| ------------------- | ------------------------------ | ------------------------------------------------------ |
| ASR partial latency | Under 500 ms                   | Starts the response while the user is still speaking   |
| LLM first token     | Under 200 ms after ASR partial | Uses prepared context                                  |
| TTS first chunk     | Under 200 ms after LLM token   | Streams speech as soon as text is available            |
| End-to-end turn     | Under 450 ms where supported   | Wall-clock target for the planned streaming-codec tier |

## Detection and routing

| Decision                | Selection rule               | Runtime behavior                                               |
| ----------------------- | ---------------------------- | -------------------------------------------------------------- |
| Audio-language modality | Model family and input shape | Runs the audio-language serving-config sweep on an audio probe |

Voice-pipeline routing resolves to the vLLM-Omni lab target, but live production execution remains blocked unless an explicit native stage config is present and accepted by the topology gate. Operators and agent runbooks pass that same-process stage config first to `scripts/plan-voice-optimization-suite.ts`, `scripts/check-voice-optimization-suite-readiness.ts`, and `scripts/hf-voice-coverage-probe.ts` with `--voice-asr-model`, `--voice-llm-model`, `--voice-tts-model`, `--voice-audio-base64-env`, and `--voice-reference-text-env`. If the paid run will use a pinned HF revision, the same `--voice-asr-revision`, `--voice-llm-revision`, and `--voice-tts-revision` values must be supplied to all three no-paid commands before any Modal work starts. The generated paid command then uses `scripts/run-optimization-lab-suite-session.ts` with the same env-backed flags and the exact env variable names supplied to planning or readiness, so agents can execute the generated command without copying raw audio or transcript payloads into default env names. Saved runbooks and agent plans do not persist raw audio or transcript payloads. When the same-process Qwen2.5-Omni pipeline session is ready, the no-paid readiness artifact must also include a Hugging Face model-intake row for `voice-pipeline-qwen-stack-l40s` with model `Qwen/Qwen2.5-Omni-7B`, expected and detected modality `audio-language`, backend `vllm-omni`, and `modelRevision` when a revision is pinned; the no-paid HF coverage report must seed the exact same Qwen model identity, including `Qwen/Qwen2.5-Omni-7B@<revision>` for pinned runs. ASR/TTS paid streak execution ignores that extra audio-language row, and suite readiness still requires it for a ready pipeline session. Current paid evidence must not fall back to the generic text ServingEngine path.

The RunInfra dashboard exposes the no-paid voice-suite readiness summary at `GET /api/optimization/lab/voice-suite/readiness`. The proxy returns the normalized Engine readiness body even when the scoped ASR/TTS gate is blocked and the HTTP status is `503`; a `200` response only means `asrTtsStreakReadiness.status` is `ready`. Before ASR/TTS paid work can start, the product path must call `POST /api/optimization/lab/voice-suite/readiness` with the no-paid `hfVoiceCoverageReport`; the app forwards it to `POST /optimization/lab/voice-suite/readiness` so Engine can bind current HF coverage, model intake, source revision, and paid-environment checks in one readiness artifact. The Optimization Lab panel renders an explicit `Voice gate` refresh for ASR, TTS, audio-language, voice-pipeline, and typed `voicePipeline` targets, and that refresh performs no paid Modal work. UI, runbook, and agent surfaces must key ASR/TTS paid-streak enablement on `asrTtsStreakReadiness`, not on the broader `suiteStatus`, because planned voice-pipeline sessions can remain blocked while the ASR/TTS five-session streak is ready.

When the active pipeline is a composed ASR -> LLM -> TTS chain and the active Test Suite has a `voice-pipeline` case with labeled audio plus reference transcript, the same POST can also include explicit `voicePipeline` config. The app proxy validates the same ASR, LLM, TTS, audio, and transcript fields that the Engine readiness route validates, then forwards them with workspace context. Readiness responses must not persist or render the raw audio payload or transcript. They may expose redacted request identity such as `audioSha256`, `audioBase64Length`, `referenceTextSha256`, and `referenceTextLength` so operators can prove the configured-pipeline blocker was cleared without leaking user audio.

Before any owner-approved paid ASR/TTS streak, operators must run the owner runner with `--verify-modal-deployment` against the same readiness artifact and cost cap. This no-paid preflight requires `MODAL_APP_URL` or `RUNINFRA_MODAL_APP_URL`, plus `COMPUTE_API_SECRET` or `RUNINFRA_COMPUTE_API_SECRET`, and calls authenticated Modal `/diagnostics/schema` before any paid child can launch. The diagnostics response must prove the deployed worker source revision, deployed worker source SHA-256, and required voice ASR/TTS diagnostics capabilities. If the runner returns `modalDeploymentVerification.status:"blocked"`, UI, runbook, and agent surfaces must treat it as a hard prelaunch blocker even when `asrTtsStreakReadiness.status` is otherwise ready. `--confirm-live` enforces the same deployed-worker verification before spawning paid children.

Before that authenticated deployment check, operators can run the no-paid public Modal health gate with `scripts/check-voice-optimization-suite-readiness.ts --verify-modal-health`. The health gate calls `MODAL_APP_URL` or `RUNINFRA_MODAL_APP_URL` at `/health` without a compute bearer and requires deployed voice capability metadata for ASR evaluation, TTS streaming and quality evaluation, speaker similarity evaluation, and voice-pipeline benchmarking. A blocked `modalHealthVerification` downgrades scoped ASR/TTS readiness with a `modal_deployment` blocker, even if the static suite plan still contains enough ready ASR/TTS sessions. As of 2026-06-24, `https://rightnow-ai--runinfra-gpu-api.modal.run/health` returned schema `2026-06-24.video-gen-benchmark-v1` and no `capabilities.voice`, so the current Modal app must be redeployed from the voice-capable `modal/unified_app.py` before any paid voice optimization or production-hosting claim.

Lab sandbox execution distinguishes measured voice quality rejection from infrastructure failure. When every Modal TTS candidate reaches the speech-quality verifier but fails the MOS or UTMOS-style floor, the Engine should persist the measured baseline and candidate rows, return `status:"quality_rejected"`, include `sandboxEvidenceId`, strict Modal call ids, and `qualityRejectionReasons`, and show the run as a quality-gate rejection. UI, runbook, and agent surfaces must not retry or label this as a generic 502 unless evidence persistence itself fails.

## Verification

TTS optimizer-trait paid-run preflight: selected TTS paid child commands that declare `--tts-family` must provide one of `f5-tts`, `cosyvoice`, `fish-speech`, `voxtral-tts`, `xtts`, `kokoro`, `style-tts`, `bark`, `parler`, `qwen3-tts`, or `unknown`. If a selected TTS paid child command declares `--tts-recommended-nfe`, it must be a positive integer. Missing or invalid optimizer-trait values block before paid Modal execution.

1. The plan should name the selected serving configuration for the audio-language model.

2. Benchmarks should report latency and throughput measured on the audio probe.

3. Voice-pipeline runs should fail closed unless the native adapter returns measured stage telemetry, including ASR, LLM, TTS, pipeline latency, and an explicit passed aggregate `voicePipelineQuality` gate. Generated voice-pipeline lab manifests must expose `experimentPlan.qualityGate.required:true`, `metric:"voicePipelineQuality"`, and `benchmarks:["WER","MOS"]` before any runbook or UI surface can present the session as quality-gated. The aggregate gate is not sufficient by itself: optimized ASR WER must not regress against baseline ASR WER, and optimized TTS MOS/UTMOS-style evidence must not regress against baseline TTS MOS/UTMOS-style evidence. Completed voice-pipeline suite-session output must project that same measured evidence into `speedup`, `quality`, `result.candidates`, and `result.winner`, including concrete ASR, LLM, and TTS Hugging Face stage model IDs. Saved certification accepts voice-pipeline latency evidence from the measured latency aliases `pipelineLatencyMs`, `voiceRoundTripMs`, `latencyP50Ms`, `latencyMs`, and `firstAudioChunkMs`; `pipelineLatencyMs` and `voiceRoundTripMs` are voice-pipeline-only aliases and cannot certify standalone ASR or TTS runs. These aliases still need measured sample-count evidence when sample counts are present, and they do not relax the stage-quality gates.

4. Production voice certification requires five consecutive passing optimization artifacts from different voice model/modality pairs, each within the 2x to 4x measured speedup band, with measured baseline quality evidence, an explicit passed optimized quality gate for ASR WER, TTS MOS/UTMOS, or aggregate voice-pipeline quality, and no optimized quality regression. Each streak artifact must carry an actual completion timestamp such as `completedAt` or `finishedAt`; `generatedAt`, `createdAt`, generic `timestamp` metadata, caller order, or filesystem order cannot certify the five-result streak. Actual completion timestamp aliases on a child output or saved artifact must parse and agree before report projection; conflicting `completedAt`, `finishedAt`, or `endedAt` aliases fail certification. Each successful paid child suite-session output must also expose actual completion timestamp evidence, and the final saved certification report must expose `summary.completionTimestampedRuns` plus a row-level actual completion timestamp such as `completedAt` on each selected passed run. The owner-paid streak runner rejects missing, under-reported, generic-only, invalid, conflicting-alias, child-Modal-call-split, or child/final timestamp-mismatched completion evidence before it can mark the run certified. Reference-voice TTS runs also require measured baseline and optimized speaker similarity, and speaker-similarity regression blocks certification. A failed run between wins resets the streak.

5. The default certification gate requires passed ASR and TTS modality coverage. Five TTS-only wins, or five ASR-only wins, do not certify the overall voice model type.

6. Paid Modal suite readiness also requires at least five distinct ready ASR/TTS model/modality sessions and must include both ASR and TTS before `paidWorkCanStart=true`. Planned but blocked voice-pipeline sessions do not satisfy this paid-start threshold. The ASR/TTS paid streak runner must also reject a certification command that requires `voice-pipeline` before launching paid children unless a real selected paid session covers `voice-pipeline`. Selected ASR paid child commands must bind WER to a supported labeled ASR eval fixture, currently `tiny-english`; unsupported fixture ids block before paid Modal execution. ASR WER threshold aliases `--asr-word-error-rate-threshold` and `--asr-max-word-error-rate` are mutually exclusive scalar flags; selected paid child commands must declare only one of them before paid Modal execution. If a selected ASR paid child command declares `--asr-num-samples`, it must be in the supported 1 to 16 range, otherwise the runner blocks before paid Modal execution. If a selected ASR paid child command declares `--asr-sample-duration-seconds`, it must be a positive integer, otherwise the runner blocks before paid Modal execution. Selected TTS paid child commands must use supported `--tts-task-type` values (`Base`, `CustomVoice`, `VoiceDesign`) and supported `--tts-speech-task` values (`base_with_reference`, `custom_voice`, `openai_voice`, `unsupported`), otherwise the suite-session CLI and paid streak runner fail closed before paid Modal execution. Selected TTS paid child commands that request speaker similarity must bind reference audio and text to a supported TTS reference fixture, currently `runinfra-reference-voice-v1`, and any selected TTS child or suite-session input that declares `--tts-reference-fixture-id` or `--tts-default-reference-fixture-id` must use that supported fixture set; unsupported reference fixture ids or invalid speaker similarity thresholds block before paid Modal execution. Selected TTS paid child commands must declare `--tts-num-samples` in the supported 1 to 16 range and must provide at least as many non-empty `--tts-text-sample` values as the declared sample count, otherwise the runner blocks before paid Modal execution. The no-paid HF coverage report must expose `coverageThresholdBasis:"supportable_hf_voice_models"`, `coverageThresholdRatio`, `overallReadyRatio`, and `coverageThresholdExcludedCount`. Operators must treat the 90 percent floor as supportable public HF voice rows only, while credential-gated, unverified, and excluded non-target rows stay auditable and cannot be presented as overall ready wins. HF rows with alternate voice checkpoint files such as Kokoro, XTTS, or MeloTTS are `support_gap` loader-adapter backlog rows until model-intake marks them ready; converted MLX, ExecuTorch, CoreML, ONNX, LiteRT, GGUF, pre-quantized, adapter-only PEFT, and empty non-standalone rows stay excluded from standalone Modal optimization coverage. Saved or replayed coverage rows must also carry row-level model id, detected modality, selected backend for ready rows, empty blockers for ready rows, and non-empty category-specific blockers for blocked rows. `support_gap` rows must keep `selectedBackend` empty until a loader adapter marks the model ready. If a selected paid child command omits the readiness-bound backend, the runner must block during model-intake and HF coverage preflight before launching that child. UI, runbook, and agent surfaces must show invalid row evidence as blocked audit/planning state, not paid-start ready state.

7. Official saved-evidence certification is source-bound. The certification command must include `--expected-source-commit-sha <40-character Engine git SHA>`, normally derived from `RUNINFRA_SOURCE_REVISION` by the voice-suite readiness report. Every generated ready session command must also include `--source-commit-sha <same Engine git SHA>` so the suite-session producer and sandbox provenance stamp matching source revision evidence such as `sourceCommitSha` or `runinfraSourceRevision` onto produced Modal artifacts. Every final row marked `passed`, including extra non-selected rows, must carry canonical `sourceCommitSha` matching the planned certification source, and that canonical row value must be a raw exact 40-character Engine SHA; whitespace-padded row values are invalid source evidence and are not trimmed into trust. Duplicate source aliases such as `source_commit_sha`, `sourceRevision`, `engineSourceCommitSha`, `runinfraSourceRevision`, `gitSha`, or `commitSha` must parse as 40-character SHAs and agree with the canonical source. The top-level `modalResultVerification` block must carry canonical `expectedSourceCommitSha` matching the planned certification source, and that canonical value must be a raw exact 40-character Engine SHA; whitespace-padded values are invalid source evidence and are not trimmed into trust. Duplicate verification source aliases such as `expected_source_commit_sha`, `sourceCommitSha`, `sourceRevision`, `engineSourceCommitSha`, `runinfraSourceRevision`, `gitSha`, or `commitSha` must parse as 40-character SHAs and agree with `expectedSourceCommitSha`. Missing, malformed, stale, mismatched, whitespace-padded, or conflicting source evidence blocks certification.

8. Voice optimization evidence is Modal-only. Each certifiable win must carry strict Modal call evidence for the measured optimization sweep: `fc-` plus 26 alphanumeric characters. Handwritten or placeholder `fc-*` strings are rejected. The same strict Modal call id cannot be repeated within a final passed row or reused across different final passed rows. Successful child suite-session output must also carry raw exact strict Modal call ids in root `modalCallIds`, `modal_call_ids`, `modalCallId`, or `modal_call_id` evidence, and selected winner/candidate Modal call-id evidence under `result`, `voiceCertification`, or `voice_certification` must be raw exact too. Whitespace-padded child stdout Modal ids fail as malformed evidence before another paid child can launch; they are not trimmed into trusted execution evidence. If a child output has malformed Modal call-id evidence and identity mismatch evidence at the same time, runbook and UI summaries should show the invalid Modal call-id stop status while retaining the identity failure details for diagnosis. Successful child suite-session modality, serving-backend, GPU, and GPU-count aliases are command identity evidence too; whitespace-padded aliases such as `modality:" asr "`, `backend:" vllm-omni "`, `gpu:" NVIDIA L4 "`, or `gpuCount:" 2 "` fail before another paid child launches instead of being trimmed into trust. The top-level `modalResultVerification.verifiedModalCallIds` block must also carry raw exact strict Modal call ids; canonical values are not trimmed or normalized, so whitespace-padded call ids fail as malformed evidence. Duplicate verification call-id aliases such as `verified_modal_call_ids`, `trustedModalCallIds`, `trusted_modal_call_ids`, `modalCallIds`, or `modal_call_ids` must be arrays of raw exact strict Modal call ids and must agree with canonical `verifiedModalCallIds`. The optimized winner must also carry winner-bound applied optimizer technique or receipt evidence; run-level optimization metadata alone is insufficient. The saved certification report exposes that accepted winner-bound evidence on each run as `optimizationTechniques`, and UI, runbook, and agent surfaces should render those values as optimization-technique proof instead of trusting a status label alone. Baseline, benchmark, measurement, manual, mock, placeholder, RunPod, or generic tool-label evidence is rejected. If baseline and optimized candidates are measured by separate Modal calls, the saved run must bind those calls in run-level `modalCallIds`; unbound split calls are rejected. When candidate-level model identity, model-revision identity, serving-backend identity, request/reference fixture identity, or hardware/load measurement identity is present, the baseline, optimized, and saved run identities must match exactly, so cross-model, cross-revision, cross-backend, cross-fixture, cross-hardware, and cross-load speedups are rejected. Explicit passed baseline and optimized ASR WER, TTS MOS/UTMOS, or aggregate voice-pipeline quality gates must share at least one benchmark identity for the same metric; cross-benchmark quality comparisons are rejected. If a quality gate or nested benchmark result cites explicit Modal call evidence, that call must be strict and bound to the same optimization sweep; unrelated quality-gate Modal calls are rejected. Official TTS suite runs synthesize three prompts by default, and the Modal result must expose `num_samples` plus MOS proxy sample coverage fields. Partial MOS or speaker-similarity coverage across generated samples blocks promotion. When measured sample counts are present on baseline or optimized latency evidence, each count must meet the minimum certification floor; undersampled speedups are rejected. Artifacts that name RunPod as the optimization provider, target, quality-gate source, or benchmark provenance, including nested benchmark result records, are rejected because RunPod belongs to BYOC deployment and canary validation, not voice optimization certification.

9. The owner-approved paid streak runner only treats final certification as certified when every successful child suite-session output has actual completion timestamp evidence, the final saved report includes summary evidence for at least five tested and passed runs, summary `testedRuns` equals the reported `runs` row count, `passedRuns`, `failedRuns`, and `passRate` match row-derived status evidence, `passedRuns + failedRuns` equals `testedRuns`, `completionTimestampedRuns` covers every tested artifact, never exceeds `testedRuns`, and matches row timestamp evidence, `uniqueModelCount`, `uniqueModelPassCount`, and `uniqueModelPassRate` match the concrete model/modality keys in reported run rows, every final row marked `passed` exposes explicit empty row-level `blockers` evidence and raw exact canonical `sourceCommitSha` matching the planned Engine source SHA with no malformed, whitespace-padded, or conflicting source aliases, row-level non-conflicting `completedAt` evidence on every selected passed run matching the corresponding child suite-session completion time, `longestConsecutivePassStreak` matches timestamp-ordered row evidence with failed rows, missing model/modality keys, and repeated model/modality keys resetting the streak, a longest consecutive pass streak of at least five, at least five unique passed models, passed ASR and TTS coverage, final summary `requiredModalities` matching the certification command's explicit `--required-modality` values or the default ASR/TTS requirement, final summary `coveredPassedModalities` and `missingRequiredModalities` matching row-derived passed modality evidence and command-required modality gaps, empty summary blockers, empty missing required modalities for the default ASR/TTS requirement, no certification-command required modality outside the selected paid-session modalities, non-empty `optimizationTechniques` on every selected passed run, and `modalResultVerification` with top-level `source` exactly `modal-result-poll`, canonical `expectedSourceCommitSha` matching the planned Engine source SHA, no malformed or conflicting Modal verification source aliases, no verification failures, no malformed or conflicting Modal verification failure aliases, strict verified Modal call ids covering every passed run, no malformed or conflicting Modal verification call-id aliases, no repeated or reused Modal call ids across final passed rows, and row-level `modalCallIds` on every final row marked `passed`, including extra non-selected rows.

10. Every final row marked `passed`, including extra non-selected rows, must carry complete row-local optimization evidence. A strict Modal call id alone is not enough, and a Modal call id already used by another final passed row cannot count as a new optimization artifact. The same row must also carry raw exact canonical source evidence matching the planned certification source, no malformed, stale, whitespace-padded, or conflicting source aliases, valid 2x to 4x speedup evidence, measured quality-preserving evidence for its concrete voice modality, recognized winner-bound optimization technique evidence, positive actual paid cost evidence, actual completion timestamp evidence, and explicit empty row blockers. Certification validator failures must surface as validation failures, not as missing-JSON parser failures, so operators can fix the real evidence defect.

## Rollback

`skipTechniques` accepts the optimizer names `quant`, `kv_cache`, `speculation`, `kernels`, and `serving`. To skip the serving-config sweep for an audio-language run:

```ts theme={"dark"}
constraints.customerOverrides = {
  skipTechniques: ["serving"],
};
```

## Availability notes

* Audio-language runs today perform a serving-config sweep on an audio probe. They do not change model weights.
* Voice-pipeline schema and HTTP routing exist. Live optimization is still disabled by default until explicit stage config is provided. The current native adapter is limited to same-model, same-revision single-process voice pipelines; arbitrary ASR plus LLM plus TTS stacks still require the co-resident multi-process worker.
* Streaming codec, cross-modal KV, encoder-precision tuning, and fully general voice-pipeline routing are planned.