Prefix caching

What it is

Prefix caching preserves KV cache blocks for prompt prefixes that recur across requests. A matching request can reuse the cached prefix and compute only the uncached suffix.

Source: vLLM, automatic prefix caching design, read 2026-09-21

Why it moves cost and latency

Reuse removes repeated prefill work for shared instructions, templates, or documents. It can reduce first-token latency and compute cost when cache hits are frequent and prefixes are long enough.

What it looks like in practice

In vLLM, block hashes cover each block and its preceding prefix, so reuse requires fully matching leading blocks. Partial prefix overlap still reuses complete matching blocks; an incomplete trailing block is not reused. Put stable content first, with gains depending on cache capacity and eviction.

Where we measured it

Related terms

Questions this definition answers