Serving
Disaggregated serving places prefill and decode work on separate worker groups. KV cache state moves between the groups when a request changes phases.
Separation lets each phase use different batching and capacity policies without competing on one worker. Cache transfer adds latency, bandwidth demand, and another partial-failure boundary.
Routing, cache format, network topology, and backpressure determine whether separation helps. Systems must recover when transfer or the destination worker fails after prefill.
Disaggregated serving places prefill and decode work on separate worker groups. KV cache state moves between the groups when a request changes phases. Routing, cache format, network topology, and backpressure determine whether separation helps. Systems must recover when transfer or the destination worker fails after prefill.
Separation lets each phase use different batching and capacity policies without competing on one worker. Cache transfer adds latency, bandwidth demand, and another partial-failure boundary.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
