Engine comparison
vLLM led measured throughput under H100 BF16, concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026: vLLM 5,333 versus TensorRT-LLM 4,813 output tokens per second. Same condition: vLLM had lower TTFT p50: vLLM 2,221 versus TensorRT-LLM 2,848 milliseconds. On H100 BF16, source cost concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026, vLLM had lower derived cost: vLLM $0.206 versus TensorRT-LLM $0.228 USD per 1M output tokens. No universal winner is claimed.
Throughput and TTFT were measured on NVIDIA H100 80GB at BF16. Cost rows are derived for the published H100 BF16, H100 FP8, and L40S BF16 configurations. The comparison uses Llama 3.1 8B Instruct, vLLM 0.23.0 and TensorRT-LLM 1.2.1, as of June 20, 2026.
Our measured latency and throughput leads change with concurrency, while backend scope limits every comparison.
We compare measured throughput and TTFT p50 at every published concurrency, then show derived cost cells at each published configuration. Every ratio sits beside both source values.
| Metric and condition | vLLM | TensorRT-LLM | Conditional verdict |
|---|---|---|---|
| Highest measured throughput on NVIDIA H100 80GB at BF16NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off | 5,333 output tokens per second | 4,813 output tokens per second | We measured higher output throughput at the highest measured point for vLLM. vLLM recorded 5,333 output tokens per second versus TensorRT-LLM at 4,813 output tokens per second, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 1.11x.derived at render time from the two measured throughput values at concurrency 256, output tokens per second, mean of three timed repeats after warmup, unique prompts with prefix caching off |
| TTFT p50 at concurrency 1NVIDIA H100 80GB, BF16, concurrency 1, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 35 milliseconds | 32 milliseconds | We measured lower TTFT p50 for TensorRT-LLM. TensorRT-LLM recorded 32 milliseconds versus vLLM at 35 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 1, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 1.09x.derived at render time from the two measured latency values at concurrency 1, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 8NVIDIA H100 80GB, BF16, concurrency 8, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 177 milliseconds | 56 milliseconds | We measured lower TTFT p50 for TensorRT-LLM. TensorRT-LLM recorded 56 milliseconds versus vLLM at 177 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 8, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 3.16x.derived at render time from the two measured latency values at concurrency 8, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 32NVIDIA H100 80GB, BF16, concurrency 32, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 514 milliseconds | 235 milliseconds | We measured lower TTFT p50 for TensorRT-LLM. TensorRT-LLM recorded 235 milliseconds versus vLLM at 514 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 32, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 2.19x.derived at render time from the two measured latency values at concurrency 32, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 64NVIDIA H100 80GB, BF16, concurrency 64, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 790 milliseconds | 504 milliseconds | We measured lower TTFT p50 for TensorRT-LLM. TensorRT-LLM recorded 504 milliseconds versus vLLM at 790 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 64, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 1.57x.derived at render time from the two measured latency values at concurrency 64, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 128NVIDIA H100 80GB, BF16, concurrency 128, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 1,658 milliseconds | 1,691 milliseconds | We measured lower TTFT p50 for vLLM. vLLM recorded 1,658 milliseconds versus TensorRT-LLM at 1,691 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 128, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 1.02x.derived at render time from the two measured latency values at concurrency 128, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| TTFT p50 at concurrency 256NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column | 2,221 milliseconds | 2,848 milliseconds | We measured lower TTFT p50 for vLLM. vLLM recorded 2,221 milliseconds versus TensorRT-LLM at 2,848 milliseconds, under NVIDIA H100 80GB, BF16, concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026. The render-derived ratio is 1.28x.derived at render time from the two measured latency values at concurrency 256, p50 time to the first streamed token, charted as the mean of three timed repeats after warmup, same request stream as the throughput column |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA H100 80GB, BF16, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026Derived from the recorded GPU rate and measured throughput at the source-selected cost concurrency, not measured directly. | $0.206 BF16, derived | $0.228 BF16, derived | We derived lower cost for vLLM. vLLM $0.206 versus TensorRT-LLM $0.228, under NVIDIA H100 80GB, BF16, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026. |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA H100 80GB, FP8, source-selected cost concurrency 256, Llama 3.1 8B Instruct, as of June 20, 2026Derived from the recorded GPU rate and measured throughput at the source-selected cost concurrency, not measured directly. | $0.158 FP8, derived | TensorRT-LLM at fp8 was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100. | We do not calculate a ratio because one published cost cell is absent. |
| USD per 1M output tokens, derived from the recorded GPU rateNVIDIA L40S 48GB, BF16, source-selected cost concurrency 128, Llama 3.1 8B Instruct, as of June 20, 2026Derived from the recorded GPU rate and measured throughput at the source-selected cost concurrency, not measured directly. | $0.425 BF16, derived | TensorRT-LLM on the L40S was not measured. The cost block publishes a single TensorRT-LLM configuration, bf16 on the H100. | We do not calculate a ratio because one published cost cell is absent. |
We keep measured throughput on its own scale. Derived cost never shares this chart.
We separate one shared prefix from many distinct prefixes because the source scopes those workloads differently.
No TensorRT-LLM prefix-cache result. Both prefix experiments ran vLLM and SGLang only.
TensorRT-LLM ran the PyTorch backend (no engine compile), so there is no compile time to report and its peak number may differ from a built TensorRT engine.
We measured throughput and TTFT only for Llama 3.1 8B Instruct on NVIDIA H100 80GB at BF16. Cost rows are limited to H100 BF16, H100 FP8, and L40S BF16. We did not test engine versions newer than vLLM 0.23.0 and TensorRT-LLM 1.2.1 on the June 20, 2026 as-of date.
Use a workspace API key and pay for input, cached input, and output tokens.
View Model APIs