Skip to main content

​
Optimization stack

TechniqueApplies whenBenefitAvailability
TEI runtimeDense encoders under 500M parametersEfficient dynamic batching for smaller embedding modelsAvailable on supported deployments
vLLM embedding routeLarger encoders and long-context embedding workloadsOpenAI-compatible /v1/embeddings with GPU batchingAvailable on supported deployments
Token-budget batchingBatch embedding trafficHigher vectors per second without changing vector semanticsAvailable on supported deployments
Small-encoder multi-replica per GPUBERT-class encoders after Modal proves distinct local replicas on one GPUHigher endpoint concurrency from one GPU without changing vector semanticsPlanned, not selectable today
Matryoshka truncationModels that explicitly declare Matryoshka supportLower storage and bandwidth with recall protectionAvailable on supported deployments
Scalar int8 vector outputMatryoshka-capable vector workloads with recall gatesLower downstream vector payload size with recall protectionCore
Binary vector output with scalar int8 rescore and float rerankVector-store workloads that support rescoring and rerankingLower index memory while preserving search qualityCore
Length-bucket batchingMixed-length RAG batchesHigher throughput for variable-length documentsCore
FP8 is not a standalone embedding quant technique. Embedding weight quantization is blocked because pooled outputs are quality-sensitive. FP8 dynamic exists only as a serving variant inside the embedding-native serving sweep, and it ships only when it clears the retrieval-quality gates below. It also requires a compatible GPU family (Hopper or Blackwell) and tends to help only on larger encoders. Pooling and normalization are not optimizations. They stay locked to the model’s training contract on every deployment, so they are listed as a correctness safeguard rather than a technique.

​
Correctness safeguards

  1. Matryoshka support is detected from model configuration, not from the model name.
  2. LLM weight-quantization methods such as AWQ and GPTQ INT4 are not used as standalone embedding optimizations.
  3. Embeddings use embedding-native serving and vector-quality sweeps rather than decode-only LLM optimizations.
  4. Pooling, normalization, and query/document prompt routing stay locked to the model’s training contract.
  5. Long-document processing avoids unsafe fixed-length truncation when the model supports longer context.
  6. Small-encoder multi-replica-per-GPU remains gated until the Modal runtime launches distinct local /v1/embeddings replicas, reports per-replica evidence, and the central recall/cosine gate passes.
  7. Decoder-style embedding repositories are admitted as embeddings only when Hugging Face metadata provides trusted embedding evidence, or when a strong embedding-family repository id is paired with decoder-family architecture proof. Broad feature-extraction metadata alone, reranker or classifier metadata, and embedding-like names with unknown custom architectures still fail closed.

​
Quality gates

MetricRequired result
cosine_drift_p99Less than 0.01
recall@10_deltaGreater than -0.005
Engine rank quality floorAt least 0.99 cosine
These gates prevent a faster embedding deployment from returning retrieval-incompatible vectors. A faster embedding variant is not promoted from inline recall telemetry alone: the central measured quality gate must run and pass, and a pending or rejected gate keeps the prior baseline active. Use qualityRegressionCeiling when your workload needs a stricter quality limit.

​
How RunInfra applies embedding optimizations

DecisionWhat RunInfra checksRuntime effect
Model admissionHugging Face task metadata, library/tags, and architecture familyRoutes trusted embedding repos, including Qwen/GTE/E5 decoder-style embedding configs, to embedding runtime support without defaulting unknown models to LLM
Runtime selectionModel size, context length, and embedding-route supportChooses the supported embedding runtime for the deployment
Token-budget batchingRequest sizes and traffic shapeSets batch and token budgets for stable throughput
Small-encoder multi-replicaBERT-class model traits plus Modal per-replica launch and recall evidenceHeld as not-yet-wired until the Modal executor reports distinct replica traffic and quality evidence
Encoder FP8 eligibilityModel traits, GPU family, and retrieval-quality gatesUses FP8 only when measured quality stays within limits
Matryoshka truncationExplicit model configuration and requested dimensionsApplies safe dimension truncation per request
Output-vector compressionRequested vector format and retrieval-quality gatesMeasures scalar int8 or binary downstream payload savings without changing the public embedding response contract. Binary candidates must pass through binary search, scalar int8 rescore, and optional float rerank gates
Pooling and normalizationModel configuration and training defaultsKeeps vector semantics stable across optimized variants
Deployment packagingWinning measured settings and any Config-tab defaultsCarries embedding-safe aliases for input length, batch caps, Matryoshka dimensions, pooling, normalization, and prompt defaults into the runtime package
Deployed inference replayLinked optimization snapshot and endpoint transportReplays selected Matryoshka dimensions and output-vector format into Modal and supported /v1/embeddings requests
Export artifactsSelected serving config and runtime backendPreserves embedding context length, batch caps, GPU utilization, pooler dimensions, pooling, normalization, and unambiguous default prompt names across shell, Docker, Compose, and Kubernetes targets
Delivery startup replaySelected serving config and runtime backendReplays selected vLLM pooler config and TEI-native pooling or prompt env into Modal endpoint, export, and RunPod BYOC launch paths. Default RunInfra Cloud embedding deploy remains blocked until certified separately.

​
Verification

  1. The optimization plan should show embedding-safe phases such as baseline profiling, embedding-native serving sweep, and deployment packaging.
  2. It should not apply decode-only LLM techniques to embedding workloads.
  3. Preview and endpoint tests should use the optimized graph or the selected deployed endpoint.
  4. Benchmark results should report vectors per second, p50 and p95 latency, VRAM, cosine quality, and recall@K when a corpus is provided.
  5. Deployment settings should match the winning measured variant shown in the dashboard.
  6. Deploy and export requests must fail closed when an optimized embedding artifact has missing, pending, or failed measured quality evidence, including quant artifacts, flat output-vector fields, and nested outputVector descriptors.
  7. If Config-tab defaults and a measured optimization both set the same embedding runtime behavior, the measured optimization wins.
  8. Launch canaries count only /v1/embeddings responses that return finite numeric vectors and, when an optimized dimension is known, the returned vector length must match it.
  9. Deployed Modal and supported /v1/embeddings inference should replay the selected optimized embedding dimension and output-vector format on every request when those settings were selected.
  10. Export artifacts should preserve selected embedding runtime settings across shell, Docker, Compose, and Kubernetes targets.
  11. Modal endpoint, export, and RunPod BYOC delivery should replay selected embedding runtime settings through startup env before certification. Default RunInfra Cloud embedding deploy remains blocked until certified separately.
  12. Before any owner-approved paid five-model embedding streak, run the owner runner with --verify-modal-deployment against the same source SHA, HF coverage report, corpus, manifest, and cost cap. This no-paid preflight requires MODAL_APP_URL or RUNINFRA_MODAL_APP_URL, plus COMPUTE_API_SECRET or RUNINFRA_COMPUTE_API_SECRET, and blocks unless authenticated Modal /diagnostics/schema proves the deployed worker source revision, deployed worker source SHA-256, and required embedding stress and quality-eval capability fields. If modalDeploymentVerification.status is blocked, no paid child, certifier, deploy, export, or runbook execution should continue.
  13. App readiness should use the same no-paid deployment preflight before a user tests embedding optimization from the RunInfra dashboard. Engine exposes GET /optimization/lab/embedding/readiness, the RunInfra dashboard proxies it at GET /api/optimization/lab/embedding/readiness, and the Optimization Lab panel mounts for embedding model nodes even when the generic dry-run panel flag is off. The panel must render blocked, unavailable, verified-but-flag-off, or paid-open states from the Engine readiness report; a blocked or unavailable report means paid embedding optimization remains closed. This check does not replace measured recall or MTEB quality gates and does not certify a five-model streak.

​
Reranker evidence notes

Reranker optimization evidence is still Modal-only. Before the first paid child can launch, the owner-approved paid rerank streak runner must verify authenticated Modal /diagnostics/schema and confirm modal_worker_source_sha256 matches the intended modal/unified_app.py source commit. Operators can run that deployed-worker check without paid GPU work by passing --verify-modal-deployment; missing or stale diagnostics must return blocked JSON with noPaidWorkPerformed:true and no child or certifier execution. Generated managed rerank smoke workloads keep a fixed 100 candidate documents across baseline and optimized variants; maxNumSeqs and maxConcurrentRequests tune scheduler batching, not workload size. Jina v3 style JinaForRanking listwise rerankers are supportable but backend-required and must not be included in TEI paid-streak candidates until TEI support or a dedicated Jina/listwise backend is wired. Rerank production canary evidence carries separate Modal completion proofs: benchmark completedAt/completed_at for throughput and qualityGateCompletedAt/quality_gate_completed_at for the quality-eval call; saved-result certification rejects fetched Modal quality-eval payloads whose completed_at does not match that quality gate timestamp. User-supplied preview or benchmark candidate lists remain capped at 500. The runner accepts nested child canary proof as either productionCanaryEvidence or production_canary_evidence, but if both aliases are present they must describe the same proof object. Conflicting nested canary evidence is a blocking audit finding before the next paid Modal child can launch. The same child output must still carry strict fc-* Modal call ids, positive measured actual cost, measured throughput, measured NDCG/MRR quality evidence, and the planned backend route. RunPod and BYOC delivery checks do not replace this Modal evidence loop.

​
Live production evidence

On 2026-06-19, capped live Modal optimization certification was run against the production Modal API endpoint for BAAI/bge-small-en-v1.5 on one NVIDIA L4, concurrency 4, duration 10s, and the tiny-retrieval embedding eval fixture. Both runs stayed under --max-estimated-cost-usd 1.
BackendStatusRequestsSuccess ratep50 latencyVectors/secPeak VRAMCost per 1K embeddingsQuality gate
vLLMsuccess51.0116.6 ms42.88161.15 GB$0.005182recall_at_2 = 1.0, threshold 0.66
TEIsuccess51.0135.4 ms36.92760.76 GB$0.006018recall_at_2 = 1.0, threshold 0.66
This evidence certifies the Modal embedding optimization path for the tested small BGE embedding model and both embedding backends. It does not certify every embedding model family or the five-model production embedding deployment streak.

​
Five-model certification status

As of 2026-06-22, the five-model embedding streak is not production-certified. The current local Engine source is a2f73f16a5bb34d9d87811b1abbb7eb9a720d498; it builds on the no-paid H100 FP8 float lane, embed-fp8-mixed, for Qwen/Qwen3-Embedding-4B, Qwen/Qwen3-Embedding-8B, BAAI/bge-multilingual-gemma2, intfloat/e5-mistral-7b-instruct, and dunzhang/stella_en_1.5B_v5. The current no-paid HF coverage report for source a2f73f16a5bb34d9d87811b1abbb7eb9a720d498 is runs/embedding-streak/hf-coverage-a2f73f16.json: status=ready, readyRatio=0.98, readyCount=49, blockedCount=1, sample.source=hf_api, and paidGpuWork:false. Coverage regeneration used current-code HF API intake for the new source. Cached HF API model-list reports older than 24 hours, invalid timestamps, or future timestamps are rejected by both coverage regeneration and paid-start command validation, so stale broad-coverage evidence cannot unlock owner-runnable commands. Capability maps before CAPABILITY_MAP_VERSION=1.3.0 are also treated as stale and recomputed, so pre-reclassification LLM or rerank maps cannot preserve old modality decisions. The only blocked row is boboliu/Qwen3-Embedding-4B-W4A16-G128, which is already packed as compressed-tensors and is held to avoid FP8 double quantization. The current public SciFact corpus is runs/embedding-streak/public-beir-scifact-test-v1.json with manifest runs/embedding-streak/public-beir-scifact-test-v1.manifest.json: 100 documents, 80 labeled queries, 8,000 query-document pairs, 94 positive labels, and SHA-256 b5b61f3ea554c12f92a6dd0c93b144cbd8695b1dac8b97dbd1891856150ca13f. The current confirm-live runner dry-run artifact is runs/embedding-streak/runner-dry-run-a2f73f16.json; it returned status:"dry_run", blockers:[], ten planned Modal commands under plan.commands, estimated cost $6.58333, zero executed runs, and noPaidWorkPerformed:true. During paid execution the runner stops before launching the next child if a successful child stdout is missing or mismatches the planned eval corpus id/SHA, the planned runtime identity, or the planned backend-capped request_texts_sha256, if it lacks a valid completion timestamp, if it lacks accepted child throughput or quality evidence, if throughput aliases conflict, if top-level quality aliases conflict with measured non-relative quality-gate aliases, if it writes anything to stderr, if it lacks coherent finite positive actual-cost aliases, if it mixes positive and nonpositive actual-cost aliases, or if it reports a measured costActualUsd above that child command’s own --max-estimated-cost-usd cap. Child capture files remain non-certifiable as runnerAccepted:false until those semantic checks pass; accepted captures are rewritten as runnerAccepted:true with canonical JSON evidence while preserving raw stdout/stderr for audit. The final runner result only reports certified when the final certification payload is a well-formed object, nested plan evidence is well-formed, source-bound to the runner plan, and matches the runner’s five planned model ids plus all ten command model, phase, candidate, estimated-cost, and displayed-command identities. That command identity gate accepts the default certifier’s serialized command string shape while rejecting altered command evidence. Final winner throughput, quality, and derived speedup must also match the accepted child stdout evidence before the runner reports certified, and child quality aliases must remain coherent with measured non-relative quality-gate values. Modal verification evidence must exist, modalVerification.verifiedModalCallIds must contain ten distinct strict fc-* call ids, modalVerification.failures must be a well-formed empty array, final evaluation blockers and winners must be well-formed arrays, final winner rows must be well-formed objects, and the saved-results certifier must return certified:true, status:"verified", zero blockers, exactly five winners, five distinct winner models, a winner model set that exactly matches the planned five models, finite winner throughput and quality metrics, reported speedup matching optimized/baseline vectors per second, speedup within 2x to 4x, and optimized quality no worse than baseline beyond the 0.005 tolerance. Saved-result certification recomputes recall over the same capped request set used by the live child (vllm 256 texts, tei 2048 texts), so capped live evidence is not compared against full-corpus embeddings. Saved-result certification also verifies fetched Modal /result completion timestamps match saved evidence exactly, requires optimized completion times to be no earlier than their paired baseline times, and still accepts wrapper-level shared completion timestamps when per-run times are unavailable. Saved-result certification also rejects conflicting throughput aliases on saved rows and fetched Modal result aliases that disagree with count/processing evidence, requires coherent finite positive measured actual-cost aliases on every saved baseline/optimized row and every fetched Modal /result, rejects mixed positive and nonpositive cost aliases, and requires fetched Modal actual cost to match local saved evidence before the call id is trusted. The Modal app used for paid certification must return matching completed_at timestamps and matching positive actual-cost evidence for source a2f73f16a5bb34d9d87811b1abbb7eb9a720d498. Production certification still requires owner approval for the paid runner, an empty results directory, an approved public SciFact proof corpus or private production corpus, the matching manifest, and saved Modal evidence showing five consecutive 2x to 4x optimized wins with no quality loss. The paid runner’s help and sample command explicitly include both --embedding-eval-corpus and --embedding-eval-corpus-manifest, so copy-pasted owner commands expose the same corpus gates enforced before paid launch. For the five-model certification path, do not use the tiny-retrieval fixture. Materialize the owner-approved retrieval set first, then pass both --embedding-eval-corpus and --embedding-eval-corpus-manifest through the planner, runner, and live certification commands. The materializer can write the required canonical corpus plus manifest from canonical JSON (--format corpus-json or --format json), combined local JSONL (--format jsonl), local BEIR-style files (--format beir --queries <queries.jsonl> --qrels <qrels.tsv>), or a bounded public BEIR SciFact subset (--public-beir scifact). The public materializer retries transient Hugging Face HTTP failures, honors Retry-After, and retries thrown fetch failures before failing closed. Public re-materialization can still be temporarily rate-limited: a rerun on 2026-06-21 hit persistent HTTP 429 after retries, but it did not overwrite the already verified corpus/manifest. The saved-result certifier verifies the manifest hash and stats, binds the Modal certification profile to the corpus id and SHA, checks request_texts_sha256 against the backend-capped request set, and recomputes recall from the fetched Modal embeddings before trusting a call id.

​
Rollback

constraints.customerOverrides = {
  skipTechniques: ["embedding-encoder-fp8-w8a8"],
};

constraints.customerOverrides = {
  qualityRegressionCeiling: 0.005,
};

​
Availability notes

  • Modal vLLM benchmark execution can run scalar int8 output-vector compression as FP16 encode plus compressed downstream index payload. Managed deployment exposure still requires compatible vector-store integration, measured recall evidence, and the selected output format being replayed by the serving path.
  • Binary output-vector compression requires compatible vector-store integration before it is shown as a managed deployment option. Binary output is certified only behind the binary search, scalar int8 rescore, and optional float rerank pipeline.
  • Small-encoder multi-replica-per-GPU is documented as a planned SOTA throughput lever, but it is not selectable or certifying until the Modal embedding worker supports replicasPerGpu, per-replica traffic evidence, and recall/cosine quality gates.
  • Corpus-aware recall@K requires a user-provided test set. Without one, RunInfra can measure throughput and cosine drift, but it will not claim retrieval recall improvement.