Memory

KV cache

What it is

The KV cache stores attention keys and values computed for earlier tokens in each active sequence. Decode steps read this state instead of recomputing the full prefix for every new token.

Why it moves cost and latency

Caching removes repeated attention work, but its memory footprint grows with sequence length and active requests. Cache pressure can cap concurrency, force eviction, or move the system into slower memory paths.

What it looks like in practice

Serving systems reserve cache capacity, allocate it as prompts and outputs grow, and release it on completion. Layout, precision, and sharing policies determine how much useful context fits on each accelerator.

Where we measured it

Related terms

Questions this definition answers

What does KV cache mean in inference serving?

The KV cache stores attention keys and values computed for earlier tokens in each active sequence. Decode steps read this state instead of recomputing the full prefix for every new token. Serving systems reserve cache capacity, allocate it as prompts and outputs grow, and release it on completion. Layout, precision, and sharing policies determine how much useful context fits on each accelerator.

Why can KV cache move cost or latency?

If you need custom optimization for a specific model, describe what you need

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

System status

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow

Memory

KV cache

What it is

The KV cache stores attention keys and values computed for earlier tokens in each active sequence. Decode steps read this state instead of recomputing the full prefix for every new token.