Audio-LM and voice-pipeline optimization runbook

​

What runs today, audio-language models

​

Voice pipeline status

​

Planned latency budget, voice pipeline

​

Detection and routing

​

Verification

1. The plan should name the selected serving configuration for the audio-language model.
2. Benchmarks should report latency and throughput measured on the audio probe.
3. Voice-pipeline runs should fail closed unless the native adapter returns measured stage telemetry, including ASR, LLM, TTS, pipeline latency, and an explicit passed aggregate voicePipelineQuality gate. Generated voice-pipeline lab manifests must expose experimentPlan.qualityGate.required:true, metric:"voicePipelineQuality", and benchmarks:["WER","MOS"] before any runbook or UI surface can present the session as quality-gated. The aggregate gate is not sufficient by itself: optimized ASR WER must not regress against baseline ASR WER, and optimized TTS MOS/UTMOS-style evidence must not regress against baseline TTS MOS/UTMOS-style evidence. Completed voice-pipeline suite-session output must project that same measured evidence into speedup, quality, result.candidates, and result.winner, including concrete ASR, LLM, and TTS Hugging Face stage model IDs. Saved certification accepts voice-pipeline latency evidence from the measured latency aliases pipelineLatencyMs, voiceRoundTripMs, latencyP50Ms, latencyMs, and firstAudioChunkMs; pipelineLatencyMs and voiceRoundTripMs are voice-pipeline-only aliases and cannot certify standalone ASR or TTS runs. These aliases still need measured sample-count evidence when sample counts are present, and they do not relax the stage-quality gates.
4. Production voice certification requires five consecutive passing optimization artifacts from different voice model/modality pairs, each within the 2x to 4x measured speedup band, with measured baseline quality evidence, an explicit passed optimized quality gate for ASR WER, TTS MOS/UTMOS, or aggregate voice-pipeline quality, and no optimized quality regression. Each streak artifact must carry an actual completion timestamp such as completedAt or finishedAt; generatedAt, createdAt, generic timestamp metadata, caller order, or filesystem order cannot certify the five-result streak. Actual completion timestamp aliases on a child output or saved artifact must parse and agree before report projection; conflicting completedAt, finishedAt, or endedAt aliases fail certification. Each successful paid child suite-session output must also expose actual completion timestamp evidence, and the final saved certification report must expose summary.completionTimestampedRuns plus a row-level actual completion timestamp such as completedAt on each selected passed run. The owner-paid streak runner rejects missing, under-reported, generic-only, invalid, conflicting-alias, child-Modal-call-split, or child/final timestamp-mismatched completion evidence before it can mark the run certified. Reference-voice TTS runs also require measured baseline and optimized speaker similarity, and speaker-similarity regression blocks certification. A failed run between wins resets the streak.
5. The default certification gate requires passed ASR and TTS modality coverage. Five TTS-only wins, or five ASR-only wins, do not certify the overall voice model type.
6. Paid Modal suite readiness also requires at least five distinct ready ASR/TTS model/modality sessions and must include both ASR and TTS before paidWorkCanStart=true. Planned but blocked voice-pipeline sessions do not satisfy this paid-start threshold. The ASR/TTS paid streak runner must also reject a certification command that requires voice-pipeline before launching paid children unless a real selected paid session covers voice-pipeline. Selected ASR paid child commands must bind WER to a supported labeled ASR eval fixture, currently tiny-english; unsupported fixture ids block before paid Modal execution. ASR WER threshold aliases --asr-word-error-rate-threshold and --asr-max-word-error-rate are mutually exclusive scalar flags; selected paid child commands must declare only one of them before paid Modal execution. If a selected ASR paid child command declares --asr-num-samples, it must be in the supported 1 to 16 range, otherwise the runner blocks before paid Modal execution. If a selected ASR paid child command declares --asr-sample-duration-seconds, it must be a positive integer, otherwise the runner blocks before paid Modal execution. Selected TTS paid child commands must use supported --tts-task-type values (Base, CustomVoice, VoiceDesign) and supported --tts-speech-task values (base_with_reference, custom_voice, openai_voice, unsupported), otherwise the suite-session CLI and paid streak runner fail closed before paid Modal execution. Selected TTS paid child commands that request speaker similarity must bind reference audio and text to a supported TTS reference fixture, currently runinfra-reference-voice-v1, and any selected TTS child or suite-session input that declares --tts-reference-fixture-id or --tts-default-reference-fixture-id must use that supported fixture set; unsupported reference fixture ids or invalid speaker similarity thresholds block before paid Modal execution. Selected TTS paid child commands must declare --tts-num-samples in the supported 1 to 16 range and must provide at least as many non-empty --tts-text-sample values as the declared sample count, otherwise the runner blocks before paid Modal execution. The no-paid HF coverage report must expose coverageThresholdBasis:"supportable_hf_voice_models", coverageThresholdRatio, overallReadyRatio, and coverageThresholdExcludedCount. Operators must treat the 90 percent floor as supportable public HF voice rows only, while credential-gated, unverified, and excluded non-target rows stay auditable and cannot be presented as overall ready wins. HF rows with alternate voice checkpoint files such as Kokoro, XTTS, or MeloTTS are support_gap loader-adapter backlog rows until model-intake marks them ready; converted MLX, ExecuTorch, CoreML, ONNX, LiteRT, GGUF, pre-quantized, adapter-only PEFT, and empty non-standalone rows stay excluded from standalone Modal optimization coverage. Saved or replayed coverage rows must also carry row-level model id, detected modality, selected backend for ready rows, empty blockers for ready rows, and non-empty category-specific blockers for blocked rows. support_gap rows must keep selectedBackend empty until a loader adapter marks the model ready. If a selected paid child command omits the readiness-bound backend, the runner must block during model-intake and HF coverage preflight before launching that child. UI, runbook, and agent surfaces must show invalid row evidence as blocked audit/planning state, not paid-start ready state.
7. Official saved-evidence certification is source-bound. The certification command must include --expected-source-commit-sha <40-character Engine git SHA>, normally derived from RUNINFRA_SOURCE_REVISION by the voice-suite readiness report. Every generated ready session command must also include --source-commit-sha <same Engine git SHA> so the suite-session producer and sandbox provenance stamp matching source revision evidence such as sourceCommitSha or runinfraSourceRevision onto produced Modal artifacts. Every final row marked passed, including extra non-selected rows, must carry canonical sourceCommitSha matching the planned certification source, and that canonical row value must be a raw exact 40-character Engine SHA; whitespace-padded row values are invalid source evidence and are not trimmed into trust. Duplicate source aliases such as source_commit_sha, sourceRevision, engineSourceCommitSha, runinfraSourceRevision, gitSha, or commitSha must parse as 40-character SHAs and agree with the canonical source. The top-level modalResultVerification block must carry canonical expectedSourceCommitSha matching the planned certification source, and that canonical value must be a raw exact 40-character Engine SHA; whitespace-padded values are invalid source evidence and are not trimmed into trust. Duplicate verification source aliases such as expected_source_commit_sha, sourceCommitSha, sourceRevision, engineSourceCommitSha, runinfraSourceRevision, gitSha, or commitSha must parse as 40-character SHAs and agree with expectedSourceCommitSha. Missing, malformed, stale, mismatched, whitespace-padded, or conflicting source evidence blocks certification.
8. Voice optimization evidence is Modal-only. Each certifiable win must carry strict Modal call evidence for the measured optimization sweep: fc- plus 26 alphanumeric characters. Handwritten or placeholder fc-* strings are rejected. The same strict Modal call id cannot be repeated within a final passed row or reused across different final passed rows. Successful child suite-session output must also carry raw exact strict Modal call ids in root modalCallIds, modal_call_ids, modalCallId, or modal_call_id evidence, and selected winner/candidate Modal call-id evidence under result, voiceCertification, or voice_certification must be raw exact too. Whitespace-padded child stdout Modal ids fail as malformed evidence before another paid child can launch; they are not trimmed into trusted execution evidence. If a child output has malformed Modal call-id evidence and identity mismatch evidence at the same time, runbook and UI summaries should show the invalid Modal call-id stop status while retaining the identity failure details for diagnosis. Successful child suite-session modality, serving-backend, GPU, and GPU-count aliases are command identity evidence too; whitespace-padded aliases such as modality:" asr ", backend:" vllm-omni ", gpu:" NVIDIA L4 ", or gpuCount:" 2 " fail before another paid child launches instead of being trimmed into trust. The top-level modalResultVerification.verifiedModalCallIds block must also carry raw exact strict Modal call ids; canonical values are not trimmed or normalized, so whitespace-padded call ids fail as malformed evidence. Duplicate verification call-id aliases such as verified_modal_call_ids, trustedModalCallIds, trusted_modal_call_ids, modalCallIds, or modal_call_ids must be arrays of raw exact strict Modal call ids and must agree with canonical verifiedModalCallIds. The optimized winner must also carry winner-bound applied optimizer technique or receipt evidence; run-level optimization metadata alone is insufficient. The saved certification report exposes that accepted winner-bound evidence on each run as optimizationTechniques, and UI, runbook, and agent surfaces should render those values as optimization-technique proof instead of trusting a status label alone. Baseline, benchmark, measurement, manual, mock, placeholder, RunPod, or generic tool-label evidence is rejected. If baseline and optimized candidates are measured by separate Modal calls, the saved run must bind those calls in run-level modalCallIds; unbound split calls are rejected. When candidate-level model identity, model-revision identity, serving-backend identity, request/reference fixture identity, or hardware/load measurement identity is present, the baseline, optimized, and saved run identities must match exactly, so cross-model, cross-revision, cross-backend, cross-fixture, cross-hardware, and cross-load speedups are rejected. Explicit passed baseline and optimized ASR WER, TTS MOS/UTMOS, or aggregate voice-pipeline quality gates must share at least one benchmark identity for the same metric; cross-benchmark quality comparisons are rejected. If a quality gate or nested benchmark result cites explicit Modal call evidence, that call must be strict and bound to the same optimization sweep; unrelated quality-gate Modal calls are rejected. Official TTS suite runs synthesize three prompts by default, and the Modal result must expose num_samples plus MOS proxy sample coverage fields. Partial MOS or speaker-similarity coverage across generated samples blocks promotion. When measured sample counts are present on baseline or optimized latency evidence, each count must meet the minimum certification floor; undersampled speedups are rejected. Artifacts that name RunPod as the optimization provider, target, quality-gate source, or benchmark provenance, including nested benchmark result records, are rejected because RunPod belongs to BYOC deployment and canary validation, not voice optimization certification.
9. The owner-approved paid streak runner only treats final certification as certified when every successful child suite-session output has actual completion timestamp evidence, the final saved report includes summary evidence for at least five tested and passed runs, summary testedRuns equals the reported runs row count, passedRuns, failedRuns, and passRate match row-derived status evidence, passedRuns + failedRuns equals testedRuns, completionTimestampedRuns covers every tested artifact, never exceeds testedRuns, and matches row timestamp evidence, uniqueModelCount, uniqueModelPassCount, and uniqueModelPassRate match the concrete model/modality keys in reported run rows, every final row marked passed exposes explicit empty row-level blockers evidence and raw exact canonical sourceCommitSha matching the planned Engine source SHA with no malformed, whitespace-padded, or conflicting source aliases, row-level non-conflicting completedAt evidence on every selected passed run matching the corresponding child suite-session completion time, longestConsecutivePassStreak matches timestamp-ordered row evidence with failed rows, missing model/modality keys, and repeated model/modality keys resetting the streak, a longest consecutive pass streak of at least five, at least five unique passed models, passed ASR and TTS coverage, final summary requiredModalities matching the certification command’s explicit --required-modality values or the default ASR/TTS requirement, final summary coveredPassedModalities and missingRequiredModalities matching row-derived passed modality evidence and command-required modality gaps, empty summary blockers, empty missing required modalities for the default ASR/TTS requirement, no certification-command required modality outside the selected paid-session modalities, non-empty optimizationTechniques on every selected passed run, and modalResultVerification with top-level source exactly modal-result-poll, canonical expectedSourceCommitSha matching the planned Engine source SHA, no malformed or conflicting Modal verification source aliases, no verification failures, no malformed or conflicting Modal verification failure aliases, strict verified Modal call ids covering every passed run, no malformed or conflicting Modal verification call-id aliases, no repeated or reused Modal call ids across final passed rows, and row-level modalCallIds on every final row marked passed, including extra non-selected rows.
10. Every final row marked passed, including extra non-selected rows, must carry complete row-local optimization evidence. A strict Modal call id alone is not enough, and a Modal call id already used by another final passed row cannot count as a new optimization artifact. The same row must also carry raw exact canonical source evidence matching the planned certification source, no malformed, stale, whitespace-padded, or conflicting source aliases, valid 2x to 4x speedup evidence, measured quality-preserving evidence for its concrete voice modality, recognized winner-bound optimization technique evidence, positive actual paid cost evidence, actual completion timestamp evidence, and explicit empty row blockers. Certification validator failures must surface as validation failures, not as missing-JSON parser failures, so operators can fix the real evidence defect.

​

Rollback

constraints.customerOverrides = {
  skipTechniques: ["serving"],
};

​

Availability notes

• Audio-language runs today perform a serving-config sweep on an audio probe. They do not change model weights.
• Voice-pipeline schema and HTTP routing exist. Live optimization is still disabled by default until explicit stage config is provided. The current native adapter is limited to same-model, same-revision single-process voice pipelines; arbitrary ASR plus LLM plus TTS stacks still require the co-resident multi-process worker.
• Streaming codec, cross-modal KV, encoder-precision tuning, and fully general voice-pipeline routing are planned.