Batching
Concurrency is the number of client requests in flight within a stated measurement boundary. It can include queued and executing work, so it is not necessarily the active model batch size.
More concurrent work can improve accelerator utilization and aggregate throughput until memory or compute saturates. Beyond that point, queueing and batch contention can raise per-request latency and tail spread.
Benchmarks hold prompt and output shapes steady while sweeping in-flight requests at the client boundary. Production systems set admission limits from cache capacity, latency objectives, and expected traffic bursts.
Concurrency is the number of client requests in flight within a stated measurement boundary. It can include queued and executing work, so it is not necessarily the active model batch size. Benchmarks hold prompt and output shapes steady while sweeping in-flight requests at the client boundary. Production systems set admission limits from cache capacity, latency objectives, and expected traffic bursts.
More concurrent work can improve accelerator utilization and aggregate throughput until memory or compute saturates. Beyond that point, queueing and batch contention can raise per-request latency and tail spread.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
