Latency
Inter-token latency is the elapsed time between consecutive generated tokens after the first token. It tracks the cadence of decode steps experienced by a request.
Slower decode cadence extends response completion time even when the first token arrives quickly. Batch contention, cache access, kernel efficiency, and cross-device communication all influence the interval.
A serving trace timestamps each streamed token and computes intervals after generation begins. Teams inspect both typical and tail intervals because pauses can hide behind a good average.
Inter-token latency is the elapsed time between consecutive generated tokens after the first token. It tracks the cadence of decode steps experienced by a request. A serving trace timestamps each streamed token and computes intervals after generation begins. Teams inspect both typical and tail intervals because pauses can hide behind a good average.
Slower decode cadence extends response completion time even when the first token arrives quickly. Batch contention, cache access, kernel efficiency, and cross-device communication all influence the interval.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
