vLLM vs SGLang vs TensorRT-LLM: a serving benchmark

Everything below is scoped to what we tested: Llama-3.1-8B-Instruct, one H100 80GB and one L40S 48GB, bf16 and fp8. We do not generalize past it. Versions, as of June 20 2026: vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1. These are self-reported results; the harness and raw data are not publicly available. Published unique-prompt throughput and latency rows cover H100 bf16. FP8 and L40S appear only in derived cost results.

The short version

There is no single fastest engine. The winner changes with how you load it.

The rule for these configs: TensorRT-LLM for latency-bound serving below its throughput ceiling, vLLM for the highest throughput and the lowest cost. SGLang tracked vLLM within a few percent.

We ran every engine on the same weights, same GPU, and same request stream. The harness and raw data are not publicly available, so these results cannot currently be independently reproduced from this publication. In the published bf16 cost rows, L40S costs about twice as much per output token as H100.

Why we ran this

Hugging Face put TGI into maintenance mode in December 2025 and archived the repository on March 21, 2026, so a lot of teams are re-choosing a serving engine right now. The comparisons you can find are point-in-time snapshots on someone else's workload. So we ran it ourselves, on our own fleet. This article records our results and their disclosed conditions; it does not provide a public reproduction harness.

How we measured

One client drives all three engines with identical request streams, so the timing definitions are the same everywhere. TTFT is the time to the first streamed token. Throughput is output tokens per second. We warm up, then time three repeats per operating point and chart the mean. Raw repeats and confidence intervals are not publicly available. Prefix caching is off for the unique-prompt sweeps so nothing gets a free cache hit.

Throughput vs concurrency

We swept concurrency from 1 to 256 at 1,024 input and 256 output tokens. At batch 1 the three engines are within four percent of each other. As load rises, TensorRT-LLM is competitive or best through concurrency 64, then flattens. vLLM keeps scaling and reaches the highest peak, 5,333 tok/s.

The latency vs throughput frontier

The throughput table hides the more useful picture. Plot first-token latency against throughput and the tradeoff is clear. TensorRT-LLM holds the lowest TTFT across the whole mid range and keeps that edge up to about 4,000 tok/s, then hits its ceiling and TTFT climbs steeply. vLLM owns the high-throughput end: it reaches 5,333 tok/s at a lower TTFT than the other two get near that rate.

So the frontier has two owners. TensorRT-LLM for latency-sensitive serving below its ceiling, vLLM for maximum throughput. That is the whole point: the right engine is a function of your operating point, not a constant.

TensorRT-LLM: the backend in version 1.2.1

Read this before you weigh the throughput numbers. We ran TensorRT-LLM on its PyTorch backend, which loads a Hugging Face checkpoint and serves it with no ahead-of-time engine build. PyTorch is the only execution backend as of TensorRT-LLM 1.2.

NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and its engine-build CLI. The 4,813 tok/s result is the highest throughput we reported for TensorRT-LLM 1.2.1 in this sweep. It does not establish a ceiling for other workloads or configurations. TensorRT-LLM had the lowest first-token latency of the three across the mid range; among the configurations we ran, vLLM reached the highest throughput.

Cost per token

We turn throughput into dollars with one formula: cost per 1M tokens equals the GPU hourly price divided by sustained tokens per hour. The H100 is $3.95 an hour and the L40S is $1.95 an hour on Modal, observed June 20 2026.

On H100, vLLM's reported derived cost is $0.158 per 1M output tokens at fp8, versus $0.206 at bf16. The published fp8 results provide no throughput or latency rows. They do not establish a speedup.

The published vLLM bf16 cost rows are $0.425 per 1M output tokens on L40S and $0.206 on H100. The L40S cost is roughly twice as high at the recorded rates and saturation points. No L40S throughput or latency rows are published, so these results do not support a speed comparison.

Shared-prefix workloads

RAG and agent traffic reuse a long system prompt across requests, so a prefix cache should help. We ran this two ways, because the shape of the reuse matters.

One shared prefix

First the simple case: one 2,048-token prefix sent to a fraction of requests, with that fraction varied from 0 to 90 percent, caching on and off. At zero hits, cache-on already exceeds cache-off: 999 vs 910 tok/s for vLLM (9.8%) and 937 vs 879 for SGLang (6.6%). This gap is unexplained by the published data and is not evidence of cache reuse. Hardware and precision for this sweep are not published. The cache-on ratios compare 90% hits with the same engine's zero-hit cache-on row, not the cache-off arm.

vLLM and SGLang were within a few percent, vLLM slightly ahead. We do not read a winner into that gap. A single shared prefix is the easy case that any block-level cache handles well. It is not the workload SGLang's RadixAttention is built for.

Many distinct prefixes, the RadixAttention case

RadixAttention is designed for many distinct prefixes held in a tree at once, the shape you get from branching conversations and a pool of system prompts. So we ran that: 256 requests spread across a growing number of distinct 2,048-token prefixes, 8 then 32 then 128 of them, both engines with caching on.

RadixAttention worked. SGLang's logs show the prefixes served straight from its tree cache. But vLLM's prefix cache stayed ahead on throughput by 6 to 8 percent at every prefix count, with lower TTFT. So on Llama-3.1-8B on one H100 we did not reproduce a RadixAttention throughput win, even on the workload it targets. Both engines cache shared prefixes well; vLLM was a few percent faster, in line with its general edge on this hardware. RadixAttention may pull ahead with longer prefixes, deeper trees, or heavier eviction pressure than we tested. On our test it did not, and we are not going to claim otherwise.

Evidence availability

The harness, raw CSVs, and confidence intervals are not publicly available. The tables preserve our self-reported measurements and disclosed conditions, but they are not a complete reproduction package. Updated September 20, 2026: the previous open-source and reproduction claims have been withdrawn.

If you would rather not run this matrix for every model you ship, that is what RunInfra does. Point it at any Hugging Face model and it benchmarks the engines, picks the config for your latency and cost target, and deploys it serverless.

FAQ

Method notes and caveats

• One H100 80GB and one L40S 48GB on Modal, single GPU, tensor-parallel size 1.
• Same Llama-3.1-8B-Instruct weights and same request stream across engines. Three timed repeats. Raw repeats and confidence intervals are not publicly available.
• Latency here is TTFT and per-request percentiles, not goodput at a fixed SLO. Whether 514 ms first-token at high load is acceptable depends on your use case; an SLO-goodput sweep is future work.
• Numbers are self-reported on our harness and are not cross-calibrated against an external suite like MLPerf. The harness and raw data are not publicly available, so this publication does not support independent reproduction.
• TensorRT-LLM 1.2.1 ran the PyTorch backend, its only execution backend. NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and engine-build CLI.
• The L40S sweep stopped at concurrency 128 (its saturation), the H100 at 256.
• Engines move weekly. These numbers are vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1, as of June 20 2026.