Glossary category
Batching glossary
Definitions for inference serving terms in the batching category.
Terms in this category
- Continuous batchingContinuous batching schedules work at each model iteration instead of waiting for an entire request batch to finish.
- ConcurrencyConcurrency is the number of client requests in flight within a stated measurement boundary.
- Chunked prefillChunked prefill divides a long prompt's prefill work into smaller token blocks.
- Continuous versus static batchingStatic batching fixes a request group until every sequence in that group finishes.
- In-flight batchingIn-flight batching is a common name for continuous batching in serving systems.
Related glossary categories
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
Ask AI about RunInfra
