Parallelism
Data-parallel serving runs independent model replicas and routes different requests to each replica. Replicas do not combine memory or compute for a single request.
More replicas scale aggregate throughput and can improve availability when placement avoids shared failures. They do not reduce one request's model latency or make an oversized model fit on one replica.
A router balances admitted requests while respecting replica health, cache locality, and adapter availability. Capacity planning includes uneven request lengths and replica startup time.
Data-parallel serving runs independent model replicas and routes different requests to each replica. Replicas do not combine memory or compute for a single request. A router balances admitted requests while respecting replica health, cache locality, and adapter availability. Capacity planning includes uneven request lengths and replica startup time.
More replicas scale aggregate throughput and can improve availability when placement avoids shared failures. They do not reduce one request's model latency or make an oversized model fit on one replica.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
