RunInfraby RightNow
  • CatalogNew
  • Pricing
  • Research
  • Contact
DashboardSign inGet started
Home/Glossary/p50 vs p99

Latency

p50 vs p99

What it is

The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution. The first locates the middle of the distribution, while the second exposes delays near the slow tail.

Why it moves cost and latency

Median latency can stay stable while a small group of requests waits much longer. Queueing, variable sequence lengths, retries, and resource contention often widen the distance between typical and tail behavior.

What it looks like in practice

Teams compute both percentiles from the same scoped request population and time window. They keep failed or cancelled requests visible in the method because filtering can make the tail look better.

Where we measured it

  • Measured median and tail latency receipts->

Related terms

  • TTFT->
  • Inter-token latency->
  • Tokens per second->
  • Concurrency->
  • Cold start->

Questions this definition answers

What does p50 vs p99 mean in inference serving?

The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution. The first locates the middle of the distribution, while the second exposes delays near the slow tail. Teams compute both percentiles from the same scoped request population and time window. They keep failed or cancelled requests visible in the method because filtering can make the tail look better.

Why can p50 vs p99 move cost or latency?

Median latency can stay stable while a small group of requests waits much longer. Queueing, variable sequence lengths, retries, and resource contention often widen the distance between typical and tail behavior.

If you need custom optimization for a specific model, describe what you need

Describe the model and hardware you want optimized...
ModelsAuto engineAuto GPU
End-to-end encryption
Isolated GPU infrastructure
No training on your data
SOC 2 Type II
RunInfraby RightNow

© 2026 RunInfra. All rights reserved.

System status
Pipeline BuilderModelsCost CalculatorPricingStartupsBenchmarksDocsResearchNewsContact
Backed by
YCombinator
AICPA Type II
SOC 2
NVIDIA Inception ProgramNVIDIA Inception Program
Ask AI about RunInfra
Part of RightNow
SecurityDPAAUPCookiesTermsPrivacy