Batching
Chunked prefill divides a long prompt's prefill work into smaller token blocks. A scheduler can place those blocks beside decode iterations instead of running the whole prompt uninterrupted.
Limiting prefill work per iteration bounds how long active decodes wait behind a large prompt. Smaller chunks improve responsiveness but may add scheduling overhead and reduce prefill efficiency.
Operators tune chunk limits with batch capacity, prompt distribution, and latency objectives. Implementations differ in how chunks share an iteration with decode work.
Chunked prefill divides a long prompt's prefill work into smaller token blocks. A scheduler can place those blocks beside decode iterations instead of running the whole prompt uninterrupted. Operators tune chunk limits with batch capacity, prompt distribution, and latency objectives. Implementations differ in how chunks share an iteration with decode work.
Limiting prefill work per iteration bounds how long active decodes wait behind a large prompt. Smaller chunks improve responsiveness but may add scheduling overhead and reduce prefill efficiency.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
