Latency
Prefill is the parallel model pass over prompt tokens that creates their KV cache state. On accelerators, sufficiently large prefills are often limited by compute throughput.
Prefill supplies much of the active model work before the first generated token. Prompt length, batching, prefix reuse, and queueing therefore shape time to first token.
Schedulers batch prompt tokens together or divide long prompts into chunks around decode work. Teams separate prefill duration from queue delay when diagnosing first-token latency.
Prefill is the parallel model pass over prompt tokens that creates their KV cache state. On accelerators, sufficiently large prefills are often limited by compute throughput. Schedulers batch prompt tokens together or divide long prompts into chunks around decode work. Teams separate prefill duration from queue delay when diagnosing first-token latency.
Prefill supplies much of the active model work before the first generated token. Prompt length, batching, prefix reuse, and queueing therefore shape time to first token.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
