Quantization

KV cache quantization

What it is

KV cache quantization stores cached attention keys and values in narrower numeric formats. Scale metadata maps the stored values back into the range expected by attention kernels.

Why it moves cost and latency

Narrower cache entries reduce memory footprint and bandwidth, which can admit more context or requests. Dequantization work and quantization error can offset those gains.

What it looks like in practice

A runtime chooses precision, scale granularity, and which cache tensors to quantize. Teams validate task quality and serving metrics against an unquantized cache.

Related terms

Questions this definition answers

What does KV cache quantization mean in inference serving?

KV cache quantization stores cached attention keys and values in narrower numeric formats. Scale metadata maps the stored values back into the range expected by attention kernels. A runtime chooses precision, scale granularity, and which cache tensors to quantize. Teams validate task quality and serving metrics against an unquantized cache.

Why can KV cache quantization move cost or latency?

Narrower cache entries reduce memory footprint and bandwidth, which can admit more context or requests. Dequantization work and quantization error can offset those gains.

KV cache quantization

What it is

Why it moves cost and latency

What it looks like in practice

Related terms

Questions this definition answers

What does KV cache quantization mean in inference serving?

Why can KV cache quantization move cost or latency?

If you need custom optimization for a specific model, describe what you need

KV cache quantization

What it is

Why it moves cost and latency

What it looks like in practice

Related terms

Questions this definition answers

What does KV cache quantization mean in inference serving?

Why can KV cache quantization move cost or latency?

If you need custom optimization for a specific model, describe what you need