Skip to main content

​
What you can override

RunInfra lets advanced users narrow the optimization search without changing the pipeline itself.
FieldTypeEffect
forceQuantstringPrefer a specific quantization mode when the selected model and runtime allow it.
skipTechniquesstring arraySkip techniques such as quantization, speculation, compile, expert-parallel, or disaggregated serving.
maxBudgetUsdnumberSet a spend ceiling for the optimization session.
targetLatencyMsnumberOverride the latency target used by the optimizer.
qualityRegressionCeilingnumberTighten or relax the maximum accepted quality regression.
hardBansarrayAdd tenant-policy bans that RunInfra must respect.

​
Safety rules

Overrides can narrow the search, but they cannot force an unsafe or incompatible deployment.
  • Hard bans still apply even when a user forces a technique.
  • If a skip list would remove every candidate, RunInfra restores a safe baseline instead of running an empty search.
  • Budget ceilings are checked before promoting an optimized version.
  • Quality gates still run before a candidate can replace the baseline.
  • Plan gates still apply. Overrides do not unlock features outside the account plan.

​
Examples

​
Disable quantization for an embedding model

constraints.customerOverrides = {
  skipTechniques: ["quant"],
};
RunInfra keeps baseline and non-quantized embedding variants in the search.

​
Prefer GPTQ INT4 for an LLM

constraints.customerOverrides = {
  forceQuant: "gptq_int4",
};
RunInfra uses the requested method only if the selected LLM and runtime support it.

​
Ban FP8 for a tenant

constraints.customerOverrides = {
  hardBans: [
    { technique: "fp8_dynamic", modality: "llm", reason: "tenant policy" },
  ],
};
Any candidate that violates the policy is excluded from the run.

​
Cap optimization spend

constraints.customerOverrides = {
  maxBudgetUsd: 5.0,
};
RunInfra treats this as a session spend ceiling, not a monthly traffic budget.

​
Audit trail

Override decisions appear in the optimization reasoning shown to the user and are preserved with the optimization record so teams can review why a candidate was included, skipped, or rejected.