> ## Documentation Index > Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Embedding optimization runbook > How RunInfra optimizes embedding deployments while preserving retrieval quality across runtime selection, batching, gated multi-replica posture, Matryoshka truncation, pooling, normalization, and recall gates. ## Optimization stack | Technique | Applies when | Benefit | Availability | | -------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ---------------------------------- | | TEI runtime | Dense encoders under 500M parameters | Efficient dynamic batching for smaller embedding models | Available on supported deployments | | vLLM embedding route | Larger encoders and long-context embedding workloads | OpenAI-compatible `/v1/embeddings` with GPU batching | Available on supported deployments | | Token-budget batching | Batch embedding traffic | Higher vectors per second without changing vector semantics | Available on supported deployments | | Small-encoder multi-replica per GPU | BERT-class encoders after Modal proves distinct local replicas on one GPU | Higher endpoint concurrency from one GPU without changing vector semantics | Planned, not selectable today | | Matryoshka truncation | Models that explicitly declare Matryoshka support | Lower storage and bandwidth with recall protection | Available on supported deployments | | Scalar int8 vector output | Matryoshka-capable vector workloads with recall gates | Lower downstream vector payload size with recall protection | Core | | Binary vector output with scalar int8 rescore and float rerank | Vector-store workloads that support rescoring and reranking | Lower index memory while preserving search quality | Core | | Length-bucket batching | Mixed-length RAG batches | Higher throughput for variable-length documents | Core | FP8 is not a standalone embedding quant technique. Embedding weight quantization is blocked because pooled outputs are quality-sensitive. FP8 dynamic exists only as a serving variant inside the embedding-native serving sweep, and it ships only when it clears the retrieval-quality gates below. It also requires a compatible GPU family (Hopper or Blackwell) and tends to help only on larger encoders. Pooling and normalization are not optimizations. They stay locked to the model's training contract on every deployment, so they are listed as a correctness safeguard rather than a technique. ## Correctness safeguards 1. Matryoshka support is detected from model configuration, not from the model name. 2. LLM weight-quantization methods such as AWQ and GPTQ INT4 are not used as standalone embedding optimizations. 3. Embeddings use embedding-native serving and vector-quality sweeps rather than decode-only LLM optimizations. 4. Pooling, normalization, and query/document prompt routing stay locked to the model's training contract. 5. Long-document processing avoids unsafe fixed-length truncation when the model supports longer context. 6. Small-encoder multi-replica-per-GPU remains gated until the Modal runtime launches distinct local `/v1/embeddings` replicas, reports per-replica evidence, and the central recall/cosine gate passes. 7. Decoder-style embedding repositories are admitted as embeddings only when Hugging Face metadata provides trusted embedding evidence, or when a strong embedding-family repository id is paired with decoder-family architecture proof. Broad `feature-extraction` metadata alone, reranker or classifier metadata, and embedding-like names with unknown custom architectures still fail closed. ## Quality gates | Metric | Required result | | ------------------------- | -------------------- | | `cosine_drift_p99` | Less than 0.01 | | `recall@10_delta` | Greater than -0.005 | | Engine rank quality floor | At least 0.99 cosine | These gates prevent a faster embedding deployment from returning retrieval-incompatible vectors. A faster embedding variant is not promoted from inline recall telemetry alone: the central measured quality gate must run and pass, and a pending or rejected gate keeps the prior baseline active. Use `qualityRegressionCeiling` when your workload needs a stricter quality limit. ## How RunInfra applies embedding optimizations | Decision | What RunInfra checks | Runtime effect | | --------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Model admission | Hugging Face task metadata, library/tags, and architecture family | Routes trusted embedding repos, including Qwen/GTE/E5 decoder-style embedding configs, to embedding runtime support without defaulting unknown models to LLM | | Runtime selection | Model size, context length, and embedding-route support | Chooses the supported embedding runtime for the deployment | | Token-budget batching | Request sizes and traffic shape | Sets batch and token budgets for stable throughput | | Small-encoder multi-replica | BERT-class model traits plus Modal per-replica launch and recall evidence | Held as not-yet-wired until the Modal executor reports distinct replica traffic and quality evidence | | Encoder FP8 eligibility | Model traits, GPU family, and retrieval-quality gates | Uses FP8 only when measured quality stays within limits | | Matryoshka truncation | Explicit model configuration and requested dimensions | Applies safe dimension truncation per request | | Output-vector compression | Requested vector format and retrieval-quality gates | Measures scalar int8 or binary downstream payload savings without changing the public embedding response contract. Binary candidates must pass through binary search, scalar int8 rescore, and optional float rerank gates | | Pooling and normalization | Model configuration and training defaults | Keeps vector semantics stable across optimized variants | | Deployment packaging | Winning measured settings and any Config-tab defaults | Carries embedding-safe aliases for input length, batch caps, Matryoshka dimensions, pooling, normalization, and prompt defaults into the runtime package | | Deployed inference replay | Linked optimization snapshot and endpoint transport | Replays selected Matryoshka dimensions and output-vector format into Modal and supported `/v1/embeddings` requests | | Export artifacts | Selected serving config and runtime backend | Preserves embedding context length, batch caps, GPU utilization, pooler dimensions, pooling, normalization, and unambiguous default prompt names across shell, Docker, Compose, and Kubernetes targets | | Delivery startup replay | Selected serving config and runtime backend | Replays selected vLLM pooler config and TEI-native pooling or prompt env into Modal endpoint, export, and RunPod BYOC launch paths. Default RunInfra Cloud embedding deploy remains blocked until certified separately. | ## Verification 1. The optimization plan should show embedding-safe phases such as baseline profiling, embedding-native serving sweep, and deployment packaging. 2. It should not apply decode-only LLM techniques to embedding workloads. 3. Preview and endpoint tests should use the optimized graph or the selected deployed endpoint. 4. Benchmark results should report vectors per second, p50 and p95 latency, VRAM, cosine quality, and recall\@K when a corpus is provided. 5. Deployment settings should match the winning measured variant shown in the dashboard. 6. Deploy and export requests must fail closed when an optimized embedding artifact has missing, pending, or failed measured quality evidence, including quant artifacts, flat output-vector fields, and nested `outputVector` descriptors. 7. If Config-tab defaults and a measured optimization both set the same embedding runtime behavior, the measured optimization wins. 8. Launch canaries count only `/v1/embeddings` responses that return finite numeric vectors and, when an optimized dimension is known, the returned vector length must match it. 9. Deployed Modal and supported `/v1/embeddings` inference should replay the selected optimized embedding dimension and output-vector format on every request when those settings were selected. 10. Export artifacts should preserve selected embedding runtime settings across shell, Docker, Compose, and Kubernetes targets. 11. Modal endpoint, export, and RunPod BYOC delivery should replay selected embedding runtime settings through startup env before certification. Default RunInfra Cloud embedding deploy remains blocked until certified separately. 12. Before any owner-approved paid five-model embedding streak, run the owner runner with `--verify-modal-deployment` against the same source SHA, HF coverage report, corpus, manifest, and cost cap. This no-paid preflight requires `MODAL_APP_URL` or `RUNINFRA_MODAL_APP_URL`, plus `COMPUTE_API_SECRET` or `RUNINFRA_COMPUTE_API_SECRET`, and blocks unless authenticated Modal `/diagnostics/schema` proves the deployed worker source revision, deployed worker source SHA-256, and required embedding stress and quality-eval capability fields. If `modalDeploymentVerification.status` is `blocked`, no paid child, certifier, deploy, export, or runbook execution should continue. 13. App readiness should use the same no-paid deployment preflight before a user tests embedding optimization from the RunInfra dashboard. Engine exposes `GET /optimization/lab/embedding/readiness`, the RunInfra dashboard proxies it at `GET /api/optimization/lab/embedding/readiness`, and the Optimization Lab panel mounts for embedding model nodes even when the generic dry-run panel flag is off. The panel must render blocked, unavailable, verified-but-flag-off, or paid-open states from the Engine readiness report; a blocked or unavailable report means paid embedding optimization remains closed. This check does not replace measured recall or MTEB quality gates and does not certify a five-model streak. ## Reranker evidence notes Reranker optimization evidence is still Modal-only. Before the first paid child can launch, the owner-approved paid rerank streak runner must verify authenticated Modal `/diagnostics/schema` and confirm `modal_worker_source_sha256` matches the intended `modal/unified_app.py` source commit. Operators can run that deployed-worker check without paid GPU work by passing `--verify-modal-deployment`; missing or stale diagnostics must return blocked JSON with `noPaidWorkPerformed:true` and no child or certifier execution. Generated managed rerank smoke workloads keep a fixed 100 candidate documents across baseline and optimized variants; `maxNumSeqs` and `maxConcurrentRequests` tune scheduler batching, not workload size. Jina v3 style `JinaForRanking` listwise rerankers are supportable but backend-required and must not be included in TEI paid-streak candidates until TEI support or a dedicated Jina/listwise backend is wired. Rerank production canary evidence carries separate Modal completion proofs: benchmark `completedAt`/`completed_at` for throughput and `qualityGateCompletedAt`/`quality_gate_completed_at` for the quality-eval call; saved-result certification rejects fetched Modal quality-eval payloads whose `completed_at` does not match that quality gate timestamp. User-supplied preview or benchmark candidate lists remain capped at 500. The runner accepts nested child canary proof as either `productionCanaryEvidence` or `production_canary_evidence`, but if both aliases are present they must describe the same proof object. Conflicting nested canary evidence is a blocking audit finding before the next paid Modal child can launch. The same child output must still carry strict `fc-*` Modal call ids, positive measured actual cost, measured throughput, measured NDCG/MRR quality evidence, and the planned backend route. RunPod and BYOC delivery checks do not replace this Modal evidence loop. ## Live production evidence On 2026-06-19, capped live Modal optimization certification was run against the production Modal API endpoint for `BAAI/bge-small-en-v1.5` on one `NVIDIA L4`, concurrency `4`, duration `10s`, and the `tiny-retrieval` embedding eval fixture. Both runs stayed under `--max-estimated-cost-usd 1`. | Backend | Status | Requests | Success rate | p50 latency | Vectors/sec | Peak VRAM | Cost per 1K embeddings | Quality gate | | ------- | ------: | -------: | -----------: | ----------: | ----------: | --------: | ---------------------: | ----------------------------------- | | vLLM | success | 5 | 1.0 | 116.6 ms | 42.8816 | 1.15 GB | \$0.005182 | recall\_at\_2 = 1.0, threshold 0.66 | | TEI | success | 5 | 1.0 | 135.4 ms | 36.9276 | 0.76 GB | \$0.006018 | recall\_at\_2 = 1.0, threshold 0.66 | This evidence certifies the Modal embedding optimization path for the tested small BGE embedding model and both embedding backends. It does not certify every embedding model family or the five-model production embedding deployment streak. ## Five-model certification status As of 2026-06-22, the five-model embedding streak is not production-certified. The current local Engine source is `a2f73f16a5bb34d9d87811b1abbb7eb9a720d498`; it builds on the no-paid H100 FP8 float lane, `embed-fp8-mixed`, for `Qwen/Qwen3-Embedding-4B`, `Qwen/Qwen3-Embedding-8B`, `BAAI/bge-multilingual-gemma2`, `intfloat/e5-mistral-7b-instruct`, and `dunzhang/stella_en_1.5B_v5`. The current no-paid HF coverage report for source `a2f73f16a5bb34d9d87811b1abbb7eb9a720d498` is `runs/embedding-streak/hf-coverage-a2f73f16.json`: `status=ready`, `readyRatio=0.98`, `readyCount=49`, `blockedCount=1`, `sample.source=hf_api`, and `paidGpuWork:false`. Coverage regeneration used current-code HF API intake for the new source. Cached HF API model-list reports older than 24 hours, invalid timestamps, or future timestamps are rejected by both coverage regeneration and paid-start command validation, so stale broad-coverage evidence cannot unlock owner-runnable commands. Capability maps before `CAPABILITY_MAP_VERSION=1.3.0` are also treated as stale and recomputed, so pre-reclassification LLM or rerank maps cannot preserve old modality decisions. The only blocked row is `boboliu/Qwen3-Embedding-4B-W4A16-G128`, which is already packed as compressed-tensors and is held to avoid FP8 double quantization. The current public SciFact corpus is `runs/embedding-streak/public-beir-scifact-test-v1.json` with manifest `runs/embedding-streak/public-beir-scifact-test-v1.manifest.json`: 100 documents, 80 labeled queries, 8,000 query-document pairs, 94 positive labels, and SHA-256 `b5b61f3ea554c12f92a6dd0c93b144cbd8695b1dac8b97dbd1891856150ca13f`. The current confirm-live runner dry-run artifact is `runs/embedding-streak/runner-dry-run-a2f73f16.json`; it returned `status:"dry_run"`, `blockers:[]`, ten planned Modal commands under `plan.commands`, estimated cost `$6.58333`, zero executed runs, and `noPaidWorkPerformed:true`. During paid execution the runner stops before launching the next child if a successful child stdout is missing or mismatches the planned eval corpus id/SHA, the planned runtime identity, or the planned backend-capped `request_texts_sha256`, if it lacks a valid completion timestamp, if it lacks accepted child throughput or quality evidence, if throughput aliases conflict, if top-level quality aliases conflict with measured non-relative quality-gate aliases, if it writes anything to stderr, if it lacks coherent finite positive actual-cost aliases, if it mixes positive and nonpositive actual-cost aliases, or if it reports a measured `costActualUsd` above that child command's own `--max-estimated-cost-usd` cap. Child capture files remain non-certifiable as `runnerAccepted:false` until those semantic checks pass; accepted captures are rewritten as `runnerAccepted:true` with canonical JSON evidence while preserving raw stdout/stderr for audit. The final runner result only reports `certified` when the final certification payload is a well-formed object, nested `plan` evidence is well-formed, source-bound to the runner plan, and matches the runner's five planned model ids plus all ten command model, phase, candidate, estimated-cost, and displayed-command identities. That command identity gate accepts the default certifier's serialized command string shape while rejecting altered command evidence. Final winner throughput, quality, and derived speedup must also match the accepted child stdout evidence before the runner reports `certified`, and child quality aliases must remain coherent with measured non-relative quality-gate values. Modal verification evidence must exist, `modalVerification.verifiedModalCallIds` must contain ten distinct strict `fc-*` call ids, `modalVerification.failures` must be a well-formed empty array, final evaluation blockers and winners must be well-formed arrays, final winner rows must be well-formed objects, and the saved-results certifier must return `certified:true`, `status:"verified"`, zero blockers, exactly five winners, five distinct winner models, a winner model set that exactly matches the planned five models, finite winner throughput and quality metrics, reported speedup matching optimized/baseline vectors per second, speedup within 2x to 4x, and optimized quality no worse than baseline beyond the 0.005 tolerance. Saved-result certification recomputes recall over the same capped request set used by the live child (`vllm` 256 texts, `tei` 2048 texts), so capped live evidence is not compared against full-corpus embeddings. Saved-result certification also verifies fetched Modal `/result` completion timestamps match saved evidence exactly, requires optimized completion times to be no earlier than their paired baseline times, and still accepts wrapper-level shared completion timestamps when per-run times are unavailable. Saved-result certification also rejects conflicting throughput aliases on saved rows and fetched Modal result aliases that disagree with count/processing evidence, requires coherent finite positive measured actual-cost aliases on every saved baseline/optimized row and every fetched Modal `/result`, rejects mixed positive and nonpositive cost aliases, and requires fetched Modal actual cost to match local saved evidence before the call id is trusted. The Modal app used for paid certification must return matching `completed_at` timestamps and matching positive actual-cost evidence for source `a2f73f16a5bb34d9d87811b1abbb7eb9a720d498`. Production certification still requires owner approval for the paid runner, an empty results directory, an approved public SciFact proof corpus or private production corpus, the matching manifest, and saved Modal evidence showing five consecutive 2x to 4x optimized wins with no quality loss. The paid runner's help and sample command explicitly include both `--embedding-eval-corpus` and `--embedding-eval-corpus-manifest`, so copy-pasted owner commands expose the same corpus gates enforced before paid launch. For the five-model certification path, do not use the `tiny-retrieval` fixture. Materialize the owner-approved retrieval set first, then pass both `--embedding-eval-corpus` and `--embedding-eval-corpus-manifest` through the planner, runner, and live certification commands. The materializer can write the required canonical corpus plus manifest from canonical JSON (`--format corpus-json` or `--format json`), combined local JSONL (`--format jsonl`), local BEIR-style files (`--format beir --queries --qrels `), or a bounded public BEIR SciFact subset (`--public-beir scifact`). The public materializer retries transient Hugging Face HTTP failures, honors `Retry-After`, and retries thrown fetch failures before failing closed. Public re-materialization can still be temporarily rate-limited: a rerun on 2026-06-21 hit persistent HTTP 429 after retries, but it did not overwrite the already verified corpus/manifest. The saved-result certifier verifies the manifest hash and stats, binds the Modal certification profile to the corpus id and SHA, checks `request_texts_sha256` against the backend-capped request set, and recomputes recall from the fetched Modal embeddings before trusting a call id. ## Rollback ```ts theme={"dark"} constraints.customerOverrides = { skipTechniques: ["embedding-encoder-fp8-w8a8"], }; constraints.customerOverrides = { qualityRegressionCeiling: 0.005, }; ``` ## Availability notes * Modal vLLM benchmark execution can run scalar int8 output-vector compression as FP16 encode plus compressed downstream index payload. Managed deployment exposure still requires compatible vector-store integration, measured recall evidence, and the selected output format being replayed by the serving path. * Binary output-vector compression requires compatible vector-store integration before it is shown as a managed deployment option. Binary output is certified only behind the binary search, scalar int8 rescore, and optional float rerank pipeline. * Small-encoder multi-replica-per-GPU is documented as a planned SOTA throughput lever, but it is not selectable or certifying until the Modal embedding worker supports `replicasPerGpu`, per-replica traffic evidence, and recall/cosine quality gates. * Corpus-aware recall\@K requires a user-provided test set. Without one, RunInfra can measure throughput and cosine drift, but it will not claim retrieval recall improvement.