Decode

What it is

Decode is the sequential generation phase after prefill, with each iteration producing one token per active sequence. It repeatedly reads model weights and growing cache state.

Why it moves cost and latency

For common serving shapes, decode is often limited by memory bandwidth rather than arithmetic throughput. Its iteration cadence drives inter-token latency and response completion time.

What it looks like in practice

A scheduler advances active sequences together while removing completions and admitting eligible work. Teams inspect token intervals, batch occupancy, cache traffic, and kernel launch overhead.

Where we measured it

Related terms

Questions this definition answers