Glossary
Inference serving glossary
Definitions for the serving mechanisms and metrics that change inference cost, latency, memory pressure, and quality.
Batching
- Continuous batchingContinuous batching schedules work at each model iteration instead of waiting for an entire request batch to finish.
- ConcurrencyConcurrency is the number of client requests in flight within a stated measurement boundary.
- Chunked prefillChunked prefill divides a long prompt's prefill work into smaller token blocks.
- Continuous versus static batchingStatic batching fixes a request group until every sequence in that group finishes.
- In-flight batchingIn-flight batching is a common name for continuous batching in serving systems.
Memory
- Paged attentionPaged attention stores each sequence's KV cache in fixed-size blocks rather than one contiguous allocation.
- KV cacheThe KV cache stores attention keys and values computed for earlier tokens in each active sequence.
- Prefix cachingPrefix caching preserves KV cache blocks for prompt prefixes that recur across requests.
- Flash attentionFlash attention is an IO-aware exact attention method that tiles computation around limited on-chip memory.
- Grouped-query attentionGrouped-query attention assigns several query heads to each shared key-value head.
- Multi-query attentionMulti-query attention gives all query heads one shared key head and one shared value head.
- Sliding-window attentionSliding-window attention limits each token to keys and values within a recent fixed-size window.
- Attention sinksAttention sinks are retained early tokens that receive attention after later tokens move through a sliding window.
- RoPE scalingRoPE scaling changes how rotary position frequencies or positions are mapped during inference.
- Context lengthContext length is the maximum number of token positions a serving configuration admits for a sequence.
Quantization
- FP8 quantizationFP8 quantization represents selected weights, activations, or cache values with eight-bit floating point formats.
- Channelwise quantizationChannelwise quantization assigns a separate scale to each channel instead of sharing one scale across an entire tensor.
- Quantization quality recoveryQuantization quality recovery is the measured retention of task performance after a model moves to lower precision.
- KV cache quantizationKV cache quantization stores cached attention keys and values in narrower numeric formats.
- AWQActivation-aware weight quantization uses activation statistics to identify weight channels that need extra protection.
- GPTQGPTQ is a layerwise post-training weight quantization method that uses approximate second-order information.
- Four-bit weight-only quantizationFour-bit weight-only quantization stores model weights as four-bit integers while keeping activations at higher precision.
- Eight-bit integer quantizationEight-bit integer quantization represents selected tensors with integer values and scale factors.
- Weight-only versus weight-activation quantizationWeight-only quantization narrows stored weights while activations remain at higher precision.
- FP8 versus eight-bit integer quantizationFP8 uses an exponent and mantissa within an eight-bit floating-point representation.
- Calibration dataCalibration data is a sample of model inputs used to derive quantization scales or protection choices.
- Outlier channelsOutlier channels contain values whose magnitudes are much larger than most peer channels.
- Mixed-precision servingMixed-precision serving assigns different numeric precisions to tensors, operations, or layers in one model configuration.
Latency
- TTFTTime to first token measures elapsed time from request arrival until the first generated token becomes available.
- Inter-token latencyInter-token latency is the elapsed time between consecutive generated tokens after the first token.
- Tokens per secondTokens per second expresses how quickly a serving system produces tokens over an interval.
- p50 vs p99The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution.
- Speculative decodingSpeculative decoding uses a cheaper draft process to propose several candidate tokens before the target model verifies them.
- PrefillPrefill is the parallel model pass over prompt tokens that creates their KV cache state.
- DecodeDecode is the sequential generation phase after prefill, with each iteration producing one token per active sequence.
- CUDA graphsCUDA graphs record a repeatable sequence of device operations and replay it with fewer host launches.
- Kernel fusionKernel fusion combines adjacent tensor operations into a single device kernel.
- Sampling parametersSampling parameters reshape the next-token distribution after model logits are computed.
Parallelism
- Tensor parallelismTensor parallelism shards weight matrices and their computation across multiple accelerators within each model layer.
- Pipeline parallelismPipeline parallelism divides consecutive model layers into stages placed on different devices.
- Tensor versus pipeline parallelismTensor parallelism splits work within model layers and exchanges partial results during those layers.
- Expert parallelismExpert parallelism places mixture-of-experts feed-forward experts on different devices.
- Data-parallel servingData-parallel serving runs independent model replicas and routes different requests to each replica.
Serving
- Disaggregated servingDisaggregated serving places prefill and decode work on separate worker groups.
- torch.compiletorch.
- Admission controlAdmission control decides which queued requests enter active model execution and which remain waiting.
- Request queueingRequest queueing is the wait between server arrival and admission to model execution.
- Streaming responsesStreaming responses send generated tokens to the caller as they become available, commonly through server-sent events.
- Offline batch inferenceOffline batch inference processes stored inputs without an interactive caller waiting for each response.
- Cold startA cold start is the delay before an inactive or new replica can accept useful model work.
- Model loadingModel loading reads checkpoint tensors and places the required weights into host or device memory.
- SafeTensorsSafeTensors is a tensor checkpoint format with a metadata header and raw tensor data.
- GGUFGGUF is a self-describing checkpoint format used widely by CPU and edge-oriented runtimes.
- Mixture-of-experts servingMixture-of-experts serving routes each token through a selected subset of available experts.
- LoRA servingLoRA serving applies low-rank adapter updates to a shared base model during inference.
- Guided decodingGuided decoding constrains generation so emitted text follows a grammar, schema, or other formal rule.
Hardware
If you need custom optimization for a specific model, describe what you need
End-to-end encryption
Isolated GPU infrastructure
No training on your data
SOC 2 Type II
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
Ask AI about RunInfra
