Loading the measured receipts and their source records.

Performance gains, with measured receipts attached.

Baseline and optimized serving measured on the same model, GPU, and workload. Every chart keeps its statistic, source record, and date in view. Cost figures are GPU compute floors derived from measured saturation, not all-in bills.

Dataset version 2026-07-12

Measured receipts at a glance

Each value stays attached to its configuration and statistic. Open the full receipt below for method, workload, quality, cost basis, and limitations.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), measured 2026-07-12

71.12 ms
Time to first token: 391.76 tok/s
Output throughput: $1.489
Per 1M output tokens: 0.973
Quality vs FP16 baseline

Source record llama31-8b-a100-2026-07-12

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), measured 2026-07-11

48.82 ms
Time to first token: 1,085 tok/s
Output throughput: $0.151
Per 1M output tokens: 0.982
Quality vs FP16 baseline

Source record qwen25-05b-t4-2026-07-11

Explore the measured results

Switch one view in place. Labels, statistic identity, source, and date stay attached to every chart.

Dataset version 2026-07-12

Request throughput, Llama 3.1 8B Instruct

NVIDIA A100 (40 GB). requests per second. higher is better.

Scroll horizontally to read every chart label and value.

Baseline FP162.24
Batched serving3.061

Source:

RunInfra measured receipt llama31-8b-a100-2026-07-12, measured 2026-07-12

What this metric means

Request throughput is the sustained number of requests the endpoint completed under the disclosed workload. Higher is better. The baseline and optimized bars use the same model, GPU, and workload, so the chart isolates the measured serving change.

Request throughput, Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB). requests per second. higher is better.

Scroll horizontally to read every chart label and value.

Baseline FP164.2
Optimized serving8.479

Source:

RunInfra measured receipt qwen25-05b-t4-2026-07-11, measured 2026-07-11

What this metric means

Read the full receipts

The complete public record behind the charts, including what was measured and what the result does not establish.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), measured 2026-07-12

Method: Continuous-batching serving tuning, plus FP8 dynamic quantization as the promoted winner, measured per configuration
Workload: RunInfra's automated load test on a single A100-40GB, identical workload on both sides. Serving statistics were measured at batch size 24.
Cost basis: A100-40GB GPU compute at the rate recorded with the run ($2.10/hr). Excludes idle time, CPU, memory, networking, and platform fees.

Baseline, FP16 serving

Request latency: 1665.58 ms
Throughput: 2.24 req/s
VRAM: 35.16 GB

RunInfra optimized, measured per configuration

Time to first token (batched serving): 71.12 ms
Time to first token (batched serving): 3472.35 ms
Inter-token latency (batched serving): 13.2 ms
Output throughput (batched serving): 391.76 tok/s
Peak VRAM (batched serving): 37 GB
Capacity (FP8 winner): 226 req/min
Cost per request (FP8 winner): $0.000155

Quality against the recorded baseline: 0.9733, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Serving-tuned (FP16, batch 24) and the promoted FP8 winner are different configurations; every number is labeled with its configuration and they are never mixed.
Time to first token p99 under saturated batch load is 3472.35 ms; the p50 to p99 spread is the cost of batching for throughput.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.

Source record llama31-8b-a100-2026-07-12, dataset 2026-07-12

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), measured 2026-07-11

Method: FP8 dynamic quantization (zero calibration) + continuous-batching serving
Workload: RunInfra's automated load test on a single T4, identical workload on both sides.
Cost basis: T4 GPU compute at the serverless rate recorded with the run ($0.59/hr). Excludes idle time, CPU, memory, networking, and platform fees.

Baseline, FP16 serving

Request latency: 793.65 ms
Request latency: 797.55 ms
Throughput: 4.2 req/s
VRAM: 11.99 GB

RunInfra optimized, FP8 dynamic + batched serving

Time to first token: 48.82 ms
Time to first token: 57.66 ms
Inter-token latency: 6.95 ms
Output throughput: 1,085 tok/s
Capacity: 509 req/min
Peak VRAM: 14.38 GB
FP8 quantized request latency: 150.23 ms

Quality against the recorded baseline: 0.9817, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.
Receipts are added only when runs pass the same provenance gate.

Source record qwen25-05b-t4-2026-07-11, dataset 2026-07-12

The measured GPU pages carry the per-package numbers

Price context, with citations

Public list prices stay separate from measured RunInfra results. Every entry keeps its original source and capture date.

Need a concrete starting point? Open the Llama 3.3 70B on H100 cost example at 50,000 requests per day. Then change the model, hardware, or traffic shape. The calculator keeps measured, cited, modeled, and absent states separate.

Serverless per-token list prices

Groq, Llama 3.1 8B Instant
$0.05 in / $0.08 out per 1M tokens
groq.com, as of 2026-07-12
DeepInfra, Llama 3.1 8B Instruct Turbo
$0.02 in / $0.03 out per 1M tokens
deepinfra.com, as of 2026-07-12
Groq, Llama 3.3 70B Versatile
$0.59 in / $0.79 out per 1M tokens
groq.com, as of 2026-07-12
DeepInfra, Llama 3.3 70B Instruct Turbo
$0.10 in / $0.32 out per 1M tokens
deepinfra.com, as of 2026-07-12
Together AI, Llama 3.3 70B
Llama 3.1 8B is no longer listed on Together's pricing page.
$1.04 in / $1.04 out per 1M tokens
www.together.ai, as of 2026-07-12

Dedicated GPU and whole-instance list prices

Together AI, H100 80GB, dedicated
$5.49 per GPU-hour
www.together.ai, as of 2026-07-12
Fireworks AI, H100 80GB, on-demand
$7.00 per GPU-hour
fireworks.ai, as of 2026-07-12
DeepInfra, H100 80GB
$2.20 per GPU-hour
deepinfra.com, as of 2026-07-12
Replicate, H100
$5.49 per GPU-hour
replicate.com, as of 2026-07-12
AWS EC2, p5.48xlarge (8x H100), us-east-1
Whole-instance SKU (about $6.88 per H100-hour as context; single H100s are not sold separately).
$55.04 per instance-hour
aws.amazon.com, as of 2026-07-12
Google Cloud, a3-highgpu-8g (8x H100), us-central1
Whole-instance SKU (about $11.06 per H100-hour as context).
$88.49 per instance-hour
cloud.google.com, as of 2026-07-12

What this page does not claim

The gaps stay visible so public pricing context is never mistaken for measured competitor performance.

We have not measured competitor latency or throughput. Provider numbers on this page are their public list prices, nothing else.
Rented per-token prices and our derived compute floor per token are different economic units (pooled utilization vs a GPU you saturate), so we never chart them on one scale.
Per-token prices across vendors are not directly comparable: tokenizers differ, and Anthropic's own pricing documentation notes its newer models' tokenizer produces roughly 30% more tokens for the same text (platform.claude.com pricing docs, as of 2026-07-12).
Only modalities with measured, provenance-checked runs appear here. More receipts are added as runs pass the gate, not before.

Dataset changelog

2026-07-12Added Llama 3.1 8B Instruct on NVIDIA A100-40GB with its 2026-07-12 measurement date and receipt record.
2026-07-12First public receipt: Qwen2.5 0.5B Instruct on NVIDIA T4, measured 2026-07-11. Provider list prices captured 2026-07-12.

Deploy your first optimized model, measured before you ship

Describe the goal. RunInfra builds and optimizes the stack.

Start Building View Pricing

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

System status

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow

Loading the measured receipts and their source records.

Performance gains, with measured receipts attached.

Dataset version 2026-07-12

Measured receipts at a glance

Each value stays attached to its configuration and statistic. Open the full receipt below for method, workload, quality, cost basis, and limitations.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), measured 2026-07-12

71.12 ms
Time to first token: 391.76 tok/s
Output throughput: $1.489
Per 1M output tokens: 0.973
Quality vs FP16 baseline

Source record llama31-8b-a100-2026-07-12

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), measured 2026-07-11

48.82 ms
Time to first token: 1,085 tok/s
Output throughput: $0.151
Per 1M output tokens: 0.982
Quality vs FP16 baseline

Source record qwen25-05b-t4-2026-07-11

Explore the measured results

Switch one view in place. Labels, statistic identity, source, and date stay attached to every chart.

Dataset version 2026-07-12

Request throughput, Llama 3.1 8B Instruct

NVIDIA A100 (40 GB). requests per second. higher is better.

Scroll horizontally to read every chart label and value.

Baseline FP162.24
Batched serving3.061

Source:

RunInfra measured receipt llama31-8b-a100-2026-07-12, measured 2026-07-12

What this metric means

Request throughput, Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB). requests per second. higher is better.

Scroll horizontally to read every chart label and value.

Baseline FP164.2
Optimized serving8.479

Source:

RunInfra measured receipt qwen25-05b-t4-2026-07-11, measured 2026-07-11

What this metric means

Read the full receipts

The complete public record behind the charts, including what was measured and what the result does not establish.

Llama 3.1 8B Instruct

NVIDIA A100 (40 GB), measured 2026-07-12

Method: Continuous-batching serving tuning, plus FP8 dynamic quantization as the promoted winner, measured per configuration
Workload: RunInfra's automated load test on a single A100-40GB, identical workload on both sides. Serving statistics were measured at batch size 24.
Cost basis: A100-40GB GPU compute at the rate recorded with the run ($2.10/hr). Excludes idle time, CPU, memory, networking, and platform fees.

Baseline, FP16 serving

Request latency: 1665.58 ms
Throughput: 2.24 req/s
VRAM: 35.16 GB

RunInfra optimized, measured per configuration

Time to first token (batched serving): 71.12 ms
Time to first token (batched serving): 3472.35 ms
Inter-token latency (batched serving): 13.2 ms
Output throughput (batched serving): 391.76 tok/s
Peak VRAM (batched serving): 37 GB
Capacity (FP8 winner): 226 req/min
Cost per request (FP8 winner): $0.000155

Quality against the recorded baseline: 0.9733, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Serving-tuned (FP16, batch 24) and the promoted FP8 winner are different configurations; every number is labeled with its configuration and they are never mixed.
Time to first token p99 under saturated batch load is 3472.35 ms; the p50 to p99 spread is the cost of batching for throughput.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.

Source record llama31-8b-a100-2026-07-12, dataset 2026-07-12

Qwen2.5 0.5B Instruct

NVIDIA T4 (16 GB), measured 2026-07-11

Method: FP8 dynamic quantization (zero calibration) + continuous-batching serving
Workload: RunInfra's automated load test on a single T4, identical workload on both sides.
Cost basis: T4 GPU compute at the serverless rate recorded with the run ($0.59/hr). Excludes idle time, CPU, memory, networking, and platform fees.

Baseline, FP16 serving

Request latency: 793.65 ms
Request latency: 797.55 ms
Throughput: 4.2 req/s
VRAM: 11.99 GB

RunInfra optimized, FP8 dynamic + batched serving

Time to first token: 48.82 ms
Time to first token: 57.66 ms
Inter-token latency: 6.95 ms
Output throughput: 1,085 tok/s
Capacity: 509 req/min
Peak VRAM: 14.38 GB
FP8 quantized request latency: 150.23 ms

Quality against the recorded baseline: 0.9817, first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline. 1.0 = no detected degradation on this composite under the disclosed workload.

Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.
Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.
Receipts are added only when runs pass the same provenance gate.

Source record qwen25-05b-t4-2026-07-11, dataset 2026-07-12

The measured GPU pages carry the per-package numbers

Price context, with citations

Public list prices stay separate from measured RunInfra results. Every entry keeps its original source and capture date.

Serverless per-token list prices

Groq, Llama 3.1 8B Instant
$0.05 in / $0.08 out per 1M tokens
groq.com, as of 2026-07-12
DeepInfra, Llama 3.1 8B Instruct Turbo
$0.02 in / $0.03 out per 1M tokens
deepinfra.com, as of 2026-07-12
Groq, Llama 3.3 70B Versatile
$0.59 in / $0.79 out per 1M tokens
groq.com, as of 2026-07-12
DeepInfra, Llama 3.3 70B Instruct Turbo
$0.10 in / $0.32 out per 1M tokens
deepinfra.com, as of 2026-07-12
Together AI, Llama 3.3 70B
Llama 3.1 8B is no longer listed on Together's pricing page.
$1.04 in / $1.04 out per 1M tokens
www.together.ai, as of 2026-07-12

Dedicated GPU and whole-instance list prices

Together AI, H100 80GB, dedicated
$5.49 per GPU-hour
www.together.ai, as of 2026-07-12
Fireworks AI, H100 80GB, on-demand
$7.00 per GPU-hour
fireworks.ai, as of 2026-07-12
DeepInfra, H100 80GB
$2.20 per GPU-hour
deepinfra.com, as of 2026-07-12
Replicate, H100
$5.49 per GPU-hour
replicate.com, as of 2026-07-12
AWS EC2, p5.48xlarge (8x H100), us-east-1
Whole-instance SKU (about $6.88 per H100-hour as context; single H100s are not sold separately).
$55.04 per instance-hour
aws.amazon.com, as of 2026-07-12
Google Cloud, a3-highgpu-8g (8x H100), us-central1
Whole-instance SKU (about $11.06 per H100-hour as context).
$88.49 per instance-hour
cloud.google.com, as of 2026-07-12

What this page does not claim

The gaps stay visible so public pricing context is never mistaken for measured competitor performance.

We have not measured competitor latency or throughput. Provider numbers on this page are their public list prices, nothing else.
Rented per-token prices and our derived compute floor per token are different economic units (pooled utilization vs a GPU you saturate), so we never chart them on one scale.
Per-token prices across vendors are not directly comparable: tokenizers differ, and Anthropic's own pricing documentation notes its newer models' tokenizer produces roughly 30% more tokens for the same text (platform.claude.com pricing docs, as of 2026-07-12).
Only modalities with measured, provenance-checked runs appear here. More receipts are added as runs pass the gate, not before.

Dataset changelog

2026-07-12Added Llama 3.1 8B Instruct on NVIDIA A100-40GB with its 2026-07-12 measurement date and receipt record.
2026-07-12First public receipt: Qwen2.5 0.5B Instruct on NVIDIA T4, measured 2026-07-11. Provider list prices captured 2026-07-12.

Deploy your first optimized model, measured before you ship

Describe the goal. RunInfra builds and optimizes the stack.

Start Building View Pricing

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

RunInfraby RightNow

System status

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow