Skip to content

RunInfraby RightNow

Optimization
Pricing
Research
Contact

Dashboard Sign in Get started

Glossary

Inference serving glossary

Definitions for the serving mechanisms and metrics that change inference cost, latency, memory pressure, and quality.

Batching

Continuous batchingContinuous batching schedules work at each model iteration instead of waiting for an entire request batch to finish.
ConcurrencyConcurrency is the number of client requests in flight within a stated measurement boundary.
Chunked prefillChunked prefill divides a long prompt's prefill work into smaller token blocks.
Continuous versus static batchingStatic batching fixes a request group until every sequence in that group finishes.
In-flight batchingIn-flight batching is a common name for continuous batching in serving systems.

Memory

Paged attentionPaged attention stores each sequence's KV cache in fixed-size blocks rather than one contiguous allocation.
KV cacheThe KV cache stores attention keys and values computed for earlier tokens in each active sequence.
Prefix cachingPrefix caching preserves KV cache blocks for prompt prefixes that recur across requests.
Flash attentionFlash attention is an IO-aware exact attention method that tiles computation around limited on-chip memory.
Grouped-query attentionGrouped-query attention assigns several query heads to each shared key-value head.
Multi-query attentionMulti-query attention gives all query heads one shared key head and one shared value head.
Sliding-window attentionSliding-window attention limits each token to keys and values within a recent fixed-size window.
Attention sinksAttention sinks are retained early tokens that receive attention after later tokens move through a sliding window.
RoPE scalingRoPE scaling changes how rotary position frequencies or positions are mapped during inference.
Context lengthContext length is the maximum number of token positions a serving configuration admits for a sequence.

Quantization

FP8 quantizationFP8 quantization represents selected weights, activations, or cache values with eight-bit floating point formats.
Channelwise quantizationChannelwise quantization assigns a separate scale to each channel instead of sharing one scale across an entire tensor.
Quantization quality recoveryQuantization quality recovery is the measured retention of task performance after a model moves to lower precision.
KV cache quantizationKV cache quantization stores cached attention keys and values in narrower numeric formats.
AWQActivation-aware weight quantization uses activation statistics to identify weight channels that need extra protection.
GPTQGPTQ is a layerwise post-training weight quantization method that uses approximate second-order information.
Four-bit weight-only quantizationFour-bit weight-only quantization stores model weights as four-bit integers while keeping activations at higher precision.
Eight-bit integer quantizationEight-bit integer quantization represents selected tensors with integer values and scale factors.
Weight-only versus weight-activation quantizationWeight-only quantization narrows stored weights while activations remain at higher precision.
FP8 versus eight-bit integer quantizationFP8 uses an exponent and mantissa within an eight-bit floating-point representation.
Calibration dataCalibration data is a sample of model inputs used to derive quantization scales or protection choices.
Outlier channelsOutlier channels contain values whose magnitudes are much larger than most peer channels.
Mixed-precision servingMixed-precision serving assigns different numeric precisions to tensors, operations, or layers in one model configuration.

Latency

TTFTTime to first token measures elapsed time from request arrival until the first generated token becomes available.
Inter-token latencyInter-token latency is the elapsed time between consecutive generated tokens after the first token.
Tokens per secondTokens per second expresses how quickly a serving system produces tokens over an interval.
p50 vs p99The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution.
Speculative decodingSpeculative decoding uses a cheaper draft process to propose several candidate tokens before the target model verifies them.
PrefillPrefill is the parallel model pass over prompt tokens that creates their KV cache state.
DecodeDecode is the sequential generation phase after prefill, with each iteration producing one token per active sequence.
CUDA graphsCUDA graphs record a repeatable sequence of device operations and replay it with fewer host launches.
Kernel fusionKernel fusion combines adjacent tensor operations into a single device kernel.
Sampling parametersSampling parameters reshape the next-token distribution after model logits are computed.

Parallelism

Tensor parallelismTensor parallelism shards weight matrices and their computation across multiple accelerators within each model layer.
Pipeline parallelismPipeline parallelism divides consecutive model layers into stages placed on different devices.
Tensor versus pipeline parallelismTensor parallelism splits work within model layers and exchanges partial results during those layers.
Expert parallelismExpert parallelism places mixture-of-experts feed-forward experts on different devices.
Data-parallel servingData-parallel serving runs independent model replicas and routes different requests to each replica.

Serving

Disaggregated servingDisaggregated serving places prefill and decode work on separate worker groups.
torch.compiletorch.compile traces supported eager-mode PyTorch execution into graphs that backends can optimize and lower to kernels.
Admission controlAdmission control decides which queued requests enter active model execution and which remain waiting.
Request queueingRequest queueing is the wait between server arrival and admission to model execution.
Streaming responsesStreaming responses send generated tokens to the caller as they become available, commonly through server-sent events.
Offline batch inferenceOffline batch inference processes stored inputs without an interactive caller waiting for each response.
Cold startA cold start is the delay before an inactive or new replica can accept useful model work.
Model loadingModel loading reads checkpoint tensors and places the required weights into host or device memory.
SafeTensorsSafeTensors is a tensor checkpoint format with a metadata header and raw tensor data.
GGUFGGUF is a self-describing checkpoint format used widely by CPU and edge-oriented runtimes.
Mixture-of-experts servingMixture-of-experts serving routes each token through a selected subset of available experts.
LoRA servingLoRA serving applies low-rank adapter updates to a shared base model during inference.
Guided decodingGuided decoding constrains generation so emitted text follows a grammar, schema, or other formal rule.

Hardware

HBMHigh-bandwidth memory is stacked device memory placed close to an accelerator's compute package.
NVLinkNVLink is a high-bandwidth connection used between supported accelerators in the same system or fabric.

Call Nemotron 3.5 Lightning 30B with Model APIs

Use a workspace API key and pay for input, cached input, and output tokens.

View Model APIs

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

© 2026 RunInfra. All rights reserved.

Join the community System status

Optimization agent Model APIs Pricing Startups Benchmarks Docs Research News Contact

Backed by

Combinator

NVIDIA Inception Program

Ask AI about RunInfra

Part of RightNow

Security DPA AUP Cookies Terms Privacy