What does p50 vs p99 mean in inference serving?
The fiftieth and ninety-ninth percentiles mark different positions in an ordered latency distribution. The first locates the middle of the distribution, while the second exposes delays near the slow tail. Teams compute both percentiles from the same scoped request population and time window. They keep failed or cancelled requests visible in the method because filtering can make the tail look better.