Latency

Tokens per second

What it is

Tokens per second expresses how quickly a serving system produces tokens over an interval. It may describe one request's generation rate or aggregate throughput across many active requests.

Why it moves cost and latency

Aggregate throughput determines how much demand a deployment can serve for a given amount of accelerator time. Per-request rate describes user-visible generation speed, so the two forms should never be mixed.

What it looks like in practice

Benchmarks state whether they count output tokens, total tokens, or per-request decode rate. They pair the rate with concurrency, prompt shape, output shape, and latency so the result remains interpretable.

Where we measured it

Related terms

Questions this definition answers

What does Tokens per second mean in inference serving?

Tokens per second expresses how quickly a serving system produces tokens over an interval. It may describe one request's generation rate or aggregate throughput across many active requests. Benchmarks state whether they count output tokens, total tokens, or per-request decode rate. They pair the rate with concurrency, prompt shape, output shape, and latency so the result remains interpretable.

Why can Tokens per second move cost or latency?

If you need custom optimization for a specific model, describe what you need

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

System status

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow

Latency

Tokens per second

What it is

Tokens per second expresses how quickly a serving system produces tokens over an interval. It may describe one request's generation rate or aggregate throughput across many active requests.