Why these exist
RunInfra optimizes different model families with different techniques. A 7B LLM benefits from serving and quantization work, an embedding model benefits from batching and dimensionality choices, and an ASR model benefits from audio-specific batching and decoding settings. Each runbook documents:- Applicable techniques: the optimization methods RunInfra can evaluate for the modality.
- Selection rules: the model, runtime, and deployment conditions that make a technique eligible.
- Runtime evidence: the launch settings, benchmark metrics, and quality checks that prove the technique actually ran.
- Rollback controls: the override that disables a technique if it hurts quality or violates a customer constraint.
Public contract
RunInfra only shows optimization work as active when it is compatible with the selected model and runtime. Modality-specific safeguards prevent LLM-only techniques from appearing as runnable work for embedding, audio, image, or video pipelines. The same compatibility rules apply across the Plan, Optimization, Code, Test, Pipeline, Deploy, Start, and Restart paths. If an older optimization record contains fields that no longer apply to the current model, those fields remain historical metadata and are not used for deployment. Optimization receipts are treated as runtime state. When a supported technique is accepted, RunInfra carries the measured configuration into generated files, preview tests, deployment payloads, and version comparison. Unsupported receipts stay visible only as audit context. Runbook executions are durable and converge honestly. An execution interrupted by a refresh, timeout, crash, or redeploy converges to ablocked, retryable state with resume and restart recovery actions; it is never left stranded as running. A reloaded session re-attaches to a running execution through a read-only execution stream, with a snapshot poll fallback, that replays phases, status, events, and persisted cost telemetry from the last delivered sequence. Cancel aborts the in-flight GPU work for the execution and finalizes its billing, and a pre-promotion cancel gate guarantees a canceled execution can never promote an optimization version afterward; work measured before the cancel remains visible as measured evidence.
Production security gates
Production database clients must pin the Supabase Postgres CA withDATABASE_SSL_CA. Deployed RunInfra dashboard and engine processes fail closed when the variable is missing, and strip Postgres SSL URL parameters such as sslmode, sslcert, sslkey, and sslrootcert before constructing the pool so the explicit pinned TLS config is authoritative.
Quant artifact runtime downloads must use RUNINFRA_RUNTIME_TOKEN_SECRET in production. OPTIMIZER_API_SECRET fallback verification is disabled unless RUNINFRA_RUNTIME_TOKEN_ALLOW_OPTIMIZER_FALLBACK=1 is set for a bounded migration window. Legacy v1 runtime tokens are also disabled in production unless RUNINFRA_RUNTIME_TOKEN_ALLOW_LEGACY_V1=1 is set. Remove both migration flags once logs no longer show fallback-secret or legacy-token matches.
Generated model-serving artifacts must not bake HF_TOKEN into Docker image layers. Gated Hugging Face model pre-downloads use Docker BuildKit secrets with docker build --secret id=HF_TOKEN,env=HF_TOKEN; runtime containers receive HF_TOKEN only through runtime environment injection when the selected model needs it.
Custom model uploads become ready only after the server confirms the S3 object exists and its byte length matches the database upload row. Client-supplied checksums are validation input only; the stored checksum is populated from S3 SHA-256 metadata when S3 exposes it.
Backend-only lab contract
RunInfra also has an internal optimization engine lab path for validating the backend before the UI is involved. The lab report is a dry-run contract: it tells operators whether a Hugging Face model, detected modality, selected backend, and optimizer lane is ready for live measurement, gated, or unsupported. The lab report does not launch Modal or RunPod work by default, does not bill credits, and does not claim optimized metrics. Itsmeasured field is false until a separate live certification run records evidence. Treat it as a readiness and evidence checklist, not a benchmark result.
Model intake is the backend-only pre-step for user-supplied Hugging Face models. POST /optimization/lab/model-intake resolves Hugging Face config metadata, detects modality, lists backend candidates, and selects a compatible lab backend when possible. If config architecture is missing or unknown, only task-specific Hugging Face metadata such as automatic-speech-recognition, text-to-speech, or audio-text-to-text can route directly to ASR, TTS, or audio-language lab paths. Broad labels such as text-generation, feature-extraction, image-to-text, and audio-to-audio require architecture or config-shape proof before the lab emits an execution manifest. Text-to-image routes to image generation only when Diffusers library, tag, or architecture evidence proves the current text-to-image runner can serve it; otherwise the task is treated as custom unsupported. Recognized but unwired tasks such as classification, rerank, and custom return explicit unsupported gates instead of falling into an unrecognized bucket. Video generation routes only through the Modal Diffusers optimization lane, and optimization promotion remains gated until the five-model consecutive 2x-4x quality-preserving Modal streak passes. Managed deploy, RunPod BYOC, and other delivery paths still require separate export and delivery certification after Modal proof; RunPod BYOC is delivery only and cannot certify optimization. Image-to-image Diffusers metadata is treated as custom unsupported until the lab has an image-input adapter and quality harness; it must not enter the current text-to-image Diffusers manifest path. POST /optimization/lab/report and POST /optimization/lab/intake also resolve through this metadata path; caller-supplied modality and backend are requested constraints, and conflicting HF metadata blocks the dry-run output instead of producing a caller-trusted report. Model intake can return a dry-run lab report, and can optionally include a cost-gated execution manifest, but it still does not queue Modal or RunPod work unless a later live runner accepts the manifest. When a live execution manifest omits optimizers, the lab defaults the accepted candidate set to the safe serving lane when serving is runnable; explicit optimizer lists remain strict, so quant, KV cache, speculation, and mixed manifests only run when requested and gated.
Reranker intake accepts HF text-ranking metadata, structured model-card Reranking results, model-card text that explicitly identifies a reranker, and decoder-only query-document relevance templates as rerank proof. This keeps BAAI bge-reranker-*, mixedbread mxbai-rerank-*, Jina sequence-classification rerankers, Alibaba GTE, and cross-encoder families on the TEI rerank path while preserving the LLM path for decoder models that only carry weak or stale ranking tags. Jina v3 style JinaForRanking listwise rerankers are recognized as rerank but remain backend-required until TEI support or a dedicated Jina/listwise backend is wired. A local live HF probe on 2026-06-20 resolved 15 representative reranker repos as rerank through both detector and config-intake paths. This is recognition and routing evidence only; production optimization certification still requires owner-approved Modal runs and the five-model streak gate below.
For vLLM rerank candidates, the dashboard treats the score method as runtime evidence. Optimization Summary, Optimization Final, and completed Runbook execution rows surface the planned score method and the Modal-observed score method when the backend returns it. Missing or drifted observed evidence is shown as a promotion blocker instead of being hidden in logs. Split benchmark verdicts are attached back to the originating apply phase row and remain scoped by both phase id and model id, so a multi-model run cannot borrow another model’s score-method proof.
Reranker runbooks expose TEI dynamic token batching, the native /rerank route, candidate-count caps, and input-length caps as separate serving techniques. Candidate count is visible planning state because Modal enforces bounded retrieved documents and saved certification rejects baseline and optimized rows with different rerank document counts.
Model intake fails closed. Gated or unreachable Hugging Face metadata returns metadata_unavailable, unsupported model families return unsupported, incompatible or unknown requested backends return blocked with valid candidates when metadata is available, and gated model weights stay blocked before live execution until the required token/access is configured. These responses are routing and readiness contracts, not measured optimization results.
POST /optimization/lab/sandbox/execute is the detached backend sandbox path for model-first execution checks before RunInfra dashboard workspace wiring is involved. It accepts the same model, mode, cost, runtime, optimizer, and TTS reference-fixture atoms as the execution boundary, but it does not require workspaceId, pipelineId, or runbook executionId, and does not assert pipeline ownership. The route delegates to the shared sandbox execution boundary, so measured-result validation matches the workspace execution path, and default public execution still starts no paid GPU work. Detached sandbox jobs and measured evidence have backend-only persistence tables, optimization_lab_sandbox_jobs and optimization_lab_sandbox_evidence, protected from public, anon, and authenticated Supabase access. Completed sandbox executions return evidenceQuery: { executionId }, which is the lookup key for GET /optimization/lab/sandbox/evidence?executionId=.... Repeating sandbox execute with an executionId that already has completed visible measured evidence returns already_completed, the existing job/evidence ids, and the same lookup key only when the stored manifest identity matches the current request before runner or billing work can start. The replay identity compares model, modality, backend, Hugging Face revision, effective GPU base/count, requested optimizer set, cost/runtime limits, test set, image-generation metadata, TTS request/profile metadata, baseline id, and candidate plan; conflicting identity returns blocked with an explicit no-paid-work message. That read endpoint returns detached sandbox evidence only when the joined sandbox job is completed and has no unresolved billing state, and returns sandbox=true, pipelineId=null, optimizationVersionId=null, measured results, result count, cost, non-billing provenance including the quality metric, source, benchmark, and testSetId, and timestamps. Missing, failed, refunded, or settlement-pending rows return 404, and database-read failures return 503; neither path performs a workspace lookup or starts runner work. The sandbox route can inject the same backend runner router only when RUNINFRA_OPTIMIZATION_LAB_SANDBOX_LIVE_RUNNER=true, Modal transport, database persistence, immutable source revision, HF_TOKEN, RUNINFRA_OPTIMIZATION_LAB_SANDBOX_MAX_COST_USD, RUNINFRA_OPTIMIZATION_LAB_SANDBOX_MAX_DURATION_SECONDS, RUNINFRA_OPTIMIZATION_LAB_SANDBOX_MAX_CONCURRENCY, and RUNINFRA_OPTIMIZATION_LAB_SANDBOX_BILLING_MODE=external_postpaid are all configured. That mode marks detached measured evidence as postpaid with explicit sandbox billing recovery metadata, does not use RunInfra dashboard workspace billing, and disables workspace-scoped agent telemetry because detached sandbox executions have no workspace row. Production detached sandbox requests must explicitly include maxCostUsd, durationSeconds, and concurrency; omitted values, over-ceiling values, unpriced GPUs, quant durations below 60 seconds, or projected GPU cost above the server/request cap return no measured evidence and an explicit no-paid-work message before Modal starts.
POST /optimization/lab/execute is the backend execution boundary. It can accept a Hugging Face model without caller-supplied modality or backend; Engine resolves HF metadata, selects a compatible backend through model intake, and blocks metadata failures, incompatible explicit backends, or caller modality mismatches before workspace, billing, or runner work starts. Caller-supplied modality and backend are treated as requested constraints, not as a bypass around Hugging Face metadata resolution. Without the live-readiness dependencies it returns runner_unavailable and starts no paid GPU work. Completed, accepted, or running scoped runbook executions can replay an existing active measured optimization version only when the stored version identity matches the current request before runner, billing, or job creation. The scoped replay identity compares model, modality, backend, Hugging Face revision, effective GPU base/count, requested optimizer set, cost/runtime limits, test set, image-generation metadata, TTS request/profile metadata, baseline id, and candidate plan; conflicting or incomplete identity returns blocked with an explicit no-paid-work message. When live execution is explicitly enabled, the current measured runner supports LLM serving sweeps for vllm, sglang, and tensorrt-llm; embedding serving sweeps for vllm and tei; built-in ASR/TTS audio fixture sweeps for vllm-omni; and ASR user-uploaded WER eval sets for vllm-omni. Other modality/backend pairs fail closed. Embedding live execution may include testSetId so the runner fetches a user-uploaded retrieval set, sends its queries to /v1/embeddings, and accepts promotion evidence only when labeled recall rows are available and the measured recall gate passes. Rerank serving promotion is held unless a relevance-judgment corpus produces measured NDCG@10 and MRR@10 inside a passed qualityGate marked measured:true; loose numeric rerank quality fields and throughput-only /rerank evidence stay pending. ASR live execution may include an asr_transcription test set with labeled audioBase64 and reference rows; the runner fetches the workspace-scoped signed URL, strips data URL prefixes, requires WAV audio with parseable duration before paid benchmarking, limits inline eval to 16 samples, and accepts promotion evidence only when measured corpus WER clears the ASR gate. TTS live execution may include a deployment-supported tts request profile with voice or paired reference audio and reference text; those request fields are part of the replay identity and flow into /v1/audio/speech sweeps while MOS proxy and transcript gates remain separate. When reference audio is present, the runner also requests measured speaker-similarity evidence from the Modal measurement path and uses the default clone threshold unless the manifest overrides it. LLM, image-generation, and other non-embedding serving live manifests with testSetId still block before queueing until each modality has wired fixture fetching and quality scoring; TTS user-uploaded test sets also remain blocked and TTS uses built-in or explicit request fixtures. Vision-language model intake can resolve compatible backends, and image-bearing chat-completions smoke is readiness metadata only; the live lab runner blocks before paid work until measured VQA, MMMU, MMBench, VLMEval, or user-uploaded multimodal quality evidence is wired. The internal RunInfra dashboard download-url call is scoped by both testSetId and workspaceId; Engine must send the caller workspace, and RunInfra dashboard must reject unscoped or cross-workspace requests before issuing any signed URL. Measured rows are persisted only after the accepted manifest, runbook execution, model/backend/step ids, bounded baseline and candidate quality metrics, backend provenance, and cost cap all match. Mixed routed runs that add paid per-runner baseline rows must include bounded quality on those additional baselines before persistence, so later runner gates cannot silently compare a candidate against itself. Mixed serving plus quant, KV cache, or speculation rows preserve step-specific quality provenance, so quant rows are labeled as quant-engine quality evidence, KV rows as KV default-prompt evidence, speculation rows as greedy-equivalence evidence, and serving rows as serving quality-gate evidence. Serving plus speculation manifests may queue only when the cent-rounded projected serving and speculation reserves both fit under maxCostUsd before any paid runner starts; quant or KV cache combined with speculation remains blocked until per-runner budget and provenance allocation is explicitly wired.
Five-model rerank streak certification requires the latest ordered streak to end with five distinct passing models, plus a production-grade relevance corpus with at least 50 documents, 10 labeled queries, 500 query-document pairs, and 10 positive labels whose document ids exist in the corpus. Tiny fixture corpora cannot emit owner-runnable Modal streak commands, and the broader tei-production engine-matrix, optimization-technique certification, direct Modal live-certifier, RunInfra Cloud delivery-smoke entrypoints, and saved evidence builders reuse the same corpus floor before paid rerank certification or production evidence can proceed. Direct Modal output summaries suppress productionCanaryEvidence for toy corpora, and RunInfra Cloud delivery-smoke summaries return a failed rerank quality gate without NDCG/MRR quality metrics when the in-memory corpus is below that floor. A corpus-backed TEI /rerank certification run also fails closed unless the dedicated Modal quality-eval worker is configured, so throughput-only benchmark success cannot be archived as measured NDCG/MRR evidence. Saved-result certification and rerank runner diagnostics prefer RERANK_MODAL_APP_URL or RUNINFRA_RERANK_MODAL_APP_URL, fall back to MODAL_APP_URL or RUNINFRA_MODAL_APP_URL, and require COMPUTE_API_SECRET or RUNINFRA_COMPUTE_API_SECRET; the Engine fetches authoritative Modal /result/{call_id} payloads and rejects results unless the quality-eval payload covers the current corpus exactly and the TEI benchmark plus quality-eval payloads match the applied TEI scheduler knobs: client batch size, batch-token budget, and concurrent-request limit. Saved-result certification treats benchmark and quality-eval completion as separate Modal proofs: benchmark completed_at must match saved run completedAt, and quality-eval completed_at must match saved production evidence qualityGateCompletedAt. If a fetched Modal wrapper contains a nested result, its wrapper status must be completed or success; pending, running, failed, canceled, cancelled, timed-out, timeout, or unknown wrapper states fail before nested metrics are trusted. TEI rerank production promotion also requires the Modal optimization path itself to meet the sustained production canary floor, currently 300 seconds and concurrency 8, plus separate RunInfra Cloud delivery proof; a short completed Modal smoke can preserve measured NDCG/MRR evidence but cannot satisfy the production gate. The Modal TEI rerank benchmark worker enforces that floor by issuing repeated concurrent native /rerank requests for the requested duration and returning observed elapsed seconds, request counts, success rate, latency percentiles, and total scored documents. The five-model streak planner defaults generated Modal commands to that 300 second and concurrency 8 floor, rejects lower CLI values before planning, requires saved production evidence to carry matching observed durationSeconds and concurrency, and direct Modal summaries below that floor report missing production proof instead of emitting productionCanaryEvidence. The planner emits baseline plus every required TEI serving candidate per model, currently rerank-throughput, rerank-scheduler-pressure, and rerank-low-latency, and includes the long-context diagnostic when the model requires it. The scheduler-pressure candidate keeps per-request document batch size moderate while raising TEI request concurrency so optimization can test scheduler backpressure separately from candidate-count caps. Saved-result certification rejects any missing planned serving candidate, any failed optimized candidate, any optimized candidate without valid authoritative Modal benchmark and quality proof, source, revision, trust-remote-code, and workload-shape evidence, and any reused optimized benchmark or quality-eval Modal call id; it then selects one valid optimized candidate as the 2x to 4x winner. Max input length remains a workload and diagnostic comparability field, not an applied TEI router override in Modal proof. Baseline and optimized rows must use distinct Modal benchmark and quality-eval call ids, so copied baseline quality proof cannot certify the optimized variant. Optional --model-revision or --hf-revision pinning is carried into each generated Modal command and rerank evidence; if revision evidence is present, baseline, optimized, saved production evidence, and authoritative Modal payloads must all match the planned HF revision. Rerankers that require custom Hugging Face code must be planned with global --trust-remote-code-required or repeated --model-trust-remote-code <model>; generated Modal commands carry --trust-remote-code-required, live load tests pass it into TEI, and saved certification rejects results unless saved output plus authoritative Modal benchmark and quality-eval payloads prove trust_remote_code=true. Baseline and optimized streak rows must compare the same rerank workload shape: same batch size, rerank document count, and maximum input length. TEI scheduler knobs such as concurrency and batch-token limits may differ as optimization knobs, but changing corpus size, max input length, HF revision, or trust-remote-code authorization invalidates the speedup proof.
Before --confirm-live emits paid TEI rerank certification commands, the five-model planner fetches current Hugging Face model metadata and requires architecture proof for every planned model. Pinned --model-revision and --hf-revision values use the Hugging Face revision endpoint, not default-branch metadata. Per Hugging Face TEI docs fetched 2026-06-21, TEI rerankers are cross-encoder or sequence-classification models and the supported reranker examples are sequence-classification families such as XLM-RoBERTa, GTE, and ModernBERT. The planner accepts TEI-compatible architecture proof such as *ForSequenceClassification and fails closed on decoder, vision-language, or listwise reranker architectures such as *ForCausalLM, *ForConditionalGeneration, and JinaForRanking until the matching backend exists. HF text-ranking recognition is therefore not enough to certify the TEI optimization path. Owner-runnable paid command planning also refuses stale or dirty source checkouts before Hugging Face metadata fetches or Modal command output: the planned --source-commit-sha must match current HEAD, and source paths under scripts, src, tests, and modal must have no tracked or untracked source changes.
GET /optimization/lab/live-readiness is the no-paid preflight for turning on the live lab runner. It performs static checks only: RUNINFRA_OPTIMIZATION_LAB_LIVE_RUNNER, RunInfra dashboard billing callback env, database persistence env, Modal measurement env, immutable Engine source revision, and optional HF_TOKEN for gated models. HTTP 200 means paidWorkCanStart=true; otherwise the route returns 503 even if static prerequisites are ready but the live runner flag is off. The local equivalent is pnpm exec tsx scripts/optimization-lab-live-readiness.ts --require-enabled. This command reads .env/.env.local, performs no network probes, starts no Modal or RunPod work, and prints env key names only, not secret values.
Voice-suite readiness for ASR, TTS, and voice-pipeline optimization is exposed to the RunInfra dashboard through GET /api/optimization/lab/voice-suite/readiness, which proxies the Engine no-paid readiness report and returns 503 whenever the scoped ASR/TTS gate or Modal deployment verification is blocked. The important contract is that the ASR/TTS Modal streak gate is scoped separately from the full suite gate, which can remain blocked because a co-located voice-pipeline session still needs evidence. UI, runbooks, and agent prompts should not collapse those two states, and must also treat modalDeploymentVerification.status:"blocked" as a hard prelaunch blocker even when ASR/TTS readiness is otherwise green. Modal is the optimization evidence path for measured ASR/TTS latency, RTF, WER, MOS proxy, and speaker-similarity gates; RunPod remains BYOC or delivery validation and is not voice optimization proof.
Dry-run reports also include an evidence block. For this contract, executionId is null, metricSource is none, measured metrics are empty, quality is not_measured, artifact provenance is not_generated, and promotion/export gates are false. A UI or client should reject any dry-run lab report that already contains measured telemetry, generated artifact ids, or an optimization version id.
Live execution evidence is read through /optimization/lab/evidence, scoped by workspace, pipeline, and durable executionId. The endpoint may surface optimized metrics only when an active optimization version, measured telemetry/result rows, and a measured quality verdict all belong to the same execution. The quality verdict carries nullable provenance fields (source, benchmark, and testSetId) so reloads can distinguish default fixtures from user-uploaded test-set scoring instead of showing a bare score. RunInfra dashboard session manifests preserve that provenance as a tab-safe quality proof label, and export manifests schema 1.2 carries the same metric, source, benchmark, and test_set_id fields in the quality block so generated eval.json and model-card text stay traceable. Export readiness is separate: it requires measured promotion plus sanitized succeeded export-job proof for the same active version. Bundle proof must have a still-valid signed URL at evidence-read time, and Hugging Face proof counts only when optimized weights were actually LFS-uploaded. Signed URLs and secrets are not returned in the evidence payload.
RunInfra dashboard may enrich that evidence with Code tab provenance from pipeline_files only when every generated file’s generationHash matches the same execution. This produces a pipeline-files:<sha256> codegen hash for Code/Pipeline readiness, but it does not unlock export readiness by itself.
RunInfra dashboard may also enrich live execution evidence with deployment provenance from deployments joined to optimization_versions. Deploy proof counts only when the row belongs to the same workspace, pipeline, durable executionId, and active optimization version, has active_verified status, and has trusted RunInfra Cloud routing metadata. Custom artifact deployments must also retain matching public gateway canary proof wherever artifact identity is present; if a row has custom artifact identity but no stored proof surface, evidence fails closed. Deploy proof does not unlock export readiness; it is the separate signal that the measured optimized version is already serving through RunInfra Cloud.
Use the lab path when refining optimization engines outside the product shell:
/api/optimization/lab/report. UI surfaces should display only readiness, blockers, and required evidence from this report. Latency, throughput, VRAM, cost, and quality should appear only after a live certification result exists.
Verification model
Every optimization should be traceable from user-visible UI to runtime evidence:- The plan names the technique and the reason it was selected.
- The benchmark response includes the effective launch settings.
- The Optimization tab displays measured latency, throughput, cost, and quality signals for the selected modality.
- Generated deployment files preserve the same runtime knobs.
- Preview tests exercise the same optimized graph the user is about to deploy.
Plan editing contract
Runbook plans stay honest under user edits:- Saving a plan edit reports whether execution actually changed. A prose-only edit saves as “Execution steps unchanged (prose only)”, a structural edit reports how many phases changed, and a label-only rename or a markdown edit that cannot be matched back onto the executable phases says so explicitly.
- Edits that would corrupt the executable phase graph are rejected with an actionable message, both at save and again at the accept gate. The rejected text stays in the editor so nothing is lost.
- Agent-authored GPU and cost lines are reconciled against the pipeline’s feasibility snapshot when one exists. A matching snapshot marks the choice as validated by feasibility data; without one, the line is visibly labeled an estimate and never renders as a validated fact.
Customer override matrix
| Field | Effect |
|---|---|
forceQuant | Override the quantization choice when the selected runtime allows it. |
skipTechniques | Skip specific technique IDs, such as speculation, quantization, compile, or prefix-cache work. |
maxBudgetUsd | Limit session spend for the optimization run. |
targetLatencyMs | Override the latency target used by the optimizer. |
qualityRegressionCeiling | Override the maximum acceptable quality drop. |
hardBans | Apply tenant-policy bans in addition to RunInfra defaults. |
Plan gating
Advanced optimization techniques depend on the plan, model family, and selected deployment runtime.| Plan | LLM | Embedding | ASR | TTS | Image generation | Vision-language |
|---|---|---|---|---|---|---|
| Free (trial) | Baseline serving sweep | Baseline embedding optimization with Modal/export/BYOC delivery; managed RunInfra Cloud deploy is gated | Baseline ASR deployment | Baseline TTS deployment | Baseline image deployment | Baseline vision-language deployment |
| Core / Enterprise | Quantization, serving, cache, and speculation work where supported | Dimension and batching optimization where supported | Audio batching and decoder optimization where supported | Streaming and voice-cache work where supported | Scheduler, precision, compile, and memory optimization where supported | Multimodal cache, concurrency, and precision work where supported |