Serving glossary

Definitions for inference serving terms in the serving category.

Terms in this category

Disaggregated servingDisaggregated serving places prefill and decode work on separate worker groups.
torch.compiletorch.compile traces supported eager-mode PyTorch execution into graphs that backends can optimize and lower to kernels.
Admission controlAdmission control decides which queued requests enter active model execution and which remain waiting.
Request queueingRequest queueing is the wait between server arrival and admission to model execution.
Streaming responsesStreaming responses send generated tokens to the caller as they become available, commonly through server-sent events.
Offline batch inferenceOffline batch inference processes stored inputs without an interactive caller waiting for each response.
Cold startA cold start is the delay before an inactive or new replica can accept useful model work.
Model loadingModel loading reads checkpoint tensors and places the required weights into host or device memory.
SafeTensorsSafeTensors is a tensor checkpoint format with a metadata header and raw tensor data.
GGUFGGUF is a self-describing checkpoint format used widely by CPU and edge-oriented runtimes.
Mixture-of-experts servingMixture-of-experts serving routes each token through a selected subset of available experts.
LoRA servingLoRA serving applies low-rank adapter updates to a shared base model during inference.
Guided decodingGuided decoding constrains generation so emitted text follows a grammar, schema, or other formal rule.

Use a workspace API key and pay for input, cached input, and output tokens.

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

Backed by

Combinator

Ask AI about RunInfra

Glossary category

Definitions for inference serving terms in the serving category.

Disaggregated servingDisaggregated serving places prefill and decode work on separate worker groups.
torch.compiletorch.compile traces supported eager-mode PyTorch execution into graphs that backends can optimize and lower to kernels.
Admission controlAdmission control decides which queued requests enter active model execution and which remain waiting.
Request queueingRequest queueing is the wait between server arrival and admission to model execution.
Streaming responsesStreaming responses send generated tokens to the caller as they become available, commonly through server-sent events.
Offline batch inferenceOffline batch inference processes stored inputs without an interactive caller waiting for each response.
Cold startA cold start is the delay before an inactive or new replica can accept useful model work.
Model loadingModel loading reads checkpoint tensors and places the required weights into host or device memory.
SafeTensorsSafeTensors is a tensor checkpoint format with a metadata header and raw tensor data.
GGUFGGUF is a self-describing checkpoint format used widely by CPU and edge-oriented runtimes.
Mixture-of-experts servingMixture-of-experts serving routes each token through a selected subset of available experts.
LoRA servingLoRA serving applies low-rank adapter updates to a shared base model during inference.
Guided decodingGuided decoding constrains generation so emitted text follows a grammar, schema, or other formal rule.