Skip to main content
Quantization makes a model smaller and usually faster, and it does so by giving up some of the model’s precision. That is a trade, not a free win. RunInfra treats it as a trade: quantization is not applied by default. A run starts from the assumption that you want the model you asked for, at full precision, served as fast as full precision can be served.

​
What runs instead

Excluding quantization does not mean excluding optimization. Everything that changes how the model is served, rather than what the model is, still runs:
  • Serving engine and runtime selection
  • Batching, chunked prefill, prefix caching, block size, and GPU memory utilization tuning
  • Lossless FP16 KV cache tuning
  • GPU selection and placement
  • Speculative decoding, which is output-identical when its equivalence check passes
These are the levers that cost you nothing in output quality, and they are searched first.

​
The four conditions that permit quantization

Quantization is enabled when at least one of these is true. Nothing else turns it on.
1

You asked for it

You requested quantization for this run, in the request itself or by selecting a quantized variant to experiment with.
2

The model does not fit the resolved GPU

The model’s base-precision weights cannot be served on the GPU the plan actually resolved. This is arithmetic, not preference: without quantization the plan does not run at all.
3

The target is local hardware

The output is aimed at a local device. Local devices are the case where base precision usually cannot be served.
4

The declared output is a constrained on-device format

The deployment target is sized for consumer devices, where base precision usually cannot be served.
A stored preference you set on the workspace always wins, in both directions. If you have explicitly allowed or denied quantization, that decision outranks all four conditions, because it is proven intent rather than an inference.

​
The fit check is a serving check, not a weights check

Condition two compares against what serving actually needs, not the raw size of the checkpoint. Serving needs the weights plus KV growth and activation headroom, so a checkpoint that is nominally smaller than a card’s memory can still fail to fit it. RunInfra uses the same fit arithmetic here that it uses to pick the GPU in the first place, so the decision to quantize can never disagree with the hardware the plan chose. If any fact needed for that judgement is missing, for example an unknown weight footprint or an unresolved GPU, the check reports that it could not evaluate and the lossless default stays. A blind spot is never converted into a reason to reduce quality.

​
The plan states the decision either way

You never have to infer what happened. Exactly one of two lines appears in the plan you read before accepting it. When quantization is in the plan, the lane carries a Quantization basis line naming the condition that permitted it. For an explicit request it reads:
For a hardware fit it names the numbers that failed, the estimated base-precision weight footprint, the VRAM actually needed once serving headroom is included, and the GPU that cannot serve it. When quantization is not in the plan, the caveats section says so and tells you how to change it:
A silently applied lossy technique is as dishonest as a silently missing one, which is why both halves of the decision are written down.

​
Asking for it

If you want quantization searched, say so:
Or give it a hardware target it has to fit, which lets the fit check earn it on the arithmetic:
Requesting quantization does not skip the accuracy check. A quantized candidate is measured first and promoted only if its quality gate passes. See Quality gates.

​
Lossy KV cache compression follows the same rule

Compressed KV cache dtypes change attention arithmetic, so they are treated as output-affecting and default to off under the same policy. When they are excluded, the run says:
The last sentence is the important one. The KV lane is not skipped, only its lossy half. Block size and prefix caching are still tuned.
  • Optimization for the full variant search and how winners are ranked.
  • Quality gates for the accuracy check a quantized candidate has to clear.
  • Run outcomes for how a run reports what it changed and what it could not measure.