{"dataset":"RunInfra optimization benchmark receipts","version":"2026-07-12","page":"https://runinfra.ai/benchmarks","receipts":[{"id":"llama31-8b-a100-2026-07-12","modelId":"meta-llama/Llama-3.1-8B-Instruct","modelLabel":"Llama 3.1 8B Instruct","hardware":"NVIDIA A100 (40 GB)","measuredAt":"2026-07-12","method":"Continuous-batching serving tuning, plus FP8 dynamic quantization as the promoted winner, measured per configuration","workload":"RunInfra's automated load test on a single A100-40GB, identical workload on both sides. Serving statistics were measured at batch size 24.","gpuBasis":"A100-40GB GPU compute at the rate recorded with the run ($2.10/hr). Excludes idle time, CPU, memory, networking, and platform fees.","charts":[{"title":"Request throughput","unit":"requests per second","direction":"higher is better","rows":[{"label":"Baseline FP16","statistic":"measured","value":2.24,"display":"2.24","tone":"neutral"},{"label":"Batched serving","statistic":"measured","value":3.061,"display":"3.061","tone":"brand"}]},{"title":"Time to first token, optimized serving","unit":"milliseconds","direction":"lower is better","rows":[{"label":"p50","statistic":"p50, measured","value":71.12,"display":"71.12","tone":"brand"},{"label":"p99","statistic":"p99, measured","value":3472.35,"display":"3,472.35","tone":"brand"}]}],"baseline":{"title":"Baseline, FP16 serving","metrics":[{"label":"Request latency","value":"1665.58 ms","statistic":"p50, measured"},{"label":"Throughput","value":"2.24 req/s","statistic":"measured"},{"label":"VRAM","value":"35.16 GB","statistic":"measured at load"}]},"optimized":{"title":"RunInfra optimized, measured per configuration","metrics":[{"label":"Time to first token (batched serving)","value":"71.12 ms","statistic":"p50, measured"},{"label":"Time to first token (batched serving)","value":"3472.35 ms","statistic":"p99, measured"},{"label":"Inter-token latency (batched serving)","value":"13.2 ms","statistic":"p50, measured"},{"label":"Output throughput (batched serving)","value":"391.76 tok/s","statistic":"measured"},{"label":"Peak VRAM (batched serving)","value":"37 GB","statistic":"measured under batch load"},{"label":"Capacity (FP8 winner)","value":"226 req/min","statistic":"measured"},{"label":"Cost per request (FP8 winner)","value":"$0.000155","statistic":"measured"}]},"headline":[{"value":"71.12 ms","label":"Time to first token","statistic":"p50, batched serving (FP16, batch 24)"},{"value":"391.76 tok/s","label":"Output throughput","statistic":"measured, batched serving (FP16, batch 24)"},{"value":"$1.489","label":"Per 1M output tokens","statistic":"GPU compute at measured saturation, batched serving config"},{"value":"0.973","label":"Quality vs FP16 baseline","statistic":"measured, FP8 winner config"}],"quality":{"score":0.9733,"evaluator":"first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline","scaleNote":"1.0 = no detected degradation on this composite under the disclosed workload"},"limitations":["Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.","Serving-tuned (FP16, batch 24) and the promoted FP8 winner are different configurations; every number is labeled with its configuration and they are never mixed.","Time to first token p99 under saturated batch load is 3472.35 ms; the p50 to p99 spread is the cost of batching for throughput.","Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees."]},{"id":"qwen25-05b-t4-2026-07-11","modelId":"Qwen/Qwen2.5-0.5B-Instruct","modelLabel":"Qwen2.5 0.5B Instruct","hardware":"NVIDIA T4 (16 GB)","measuredAt":"2026-07-11","method":"FP8 dynamic quantization (zero calibration) + continuous-batching serving","workload":"RunInfra's automated load test on a single T4, identical workload on both sides.","gpuBasis":"T4 GPU compute at the serverless rate recorded with the run ($0.59/hr). Excludes idle time, CPU, memory, networking, and platform fees.","charts":[{"title":"Request throughput","unit":"requests per second","direction":"higher is better","rows":[{"label":"Baseline FP16","statistic":"measured","value":4.2,"display":"4.2","tone":"neutral"},{"label":"Optimized serving","statistic":"measured","value":8.479,"display":"8.479","tone":"brand"}]},{"title":"Time to first token, optimized serving","unit":"milliseconds","direction":"lower is better","rows":[{"label":"p50","statistic":"p50, measured","value":48.82,"display":"48.82","tone":"brand"},{"label":"p99","statistic":"p99, measured","value":57.66,"display":"57.66","tone":"brand"}]}],"baseline":{"title":"Baseline, FP16 serving","metrics":[{"label":"Request latency","value":"793.65 ms","statistic":"p50, measured"},{"label":"Request latency","value":"797.55 ms","statistic":"p99, measured"},{"label":"Throughput","value":"4.2 req/s","statistic":"measured"},{"label":"VRAM","value":"11.99 GB","statistic":"measured at load"}]},"optimized":{"title":"RunInfra optimized, FP8 dynamic + batched serving","metrics":[{"label":"Time to first token","value":"48.82 ms","statistic":"p50, measured"},{"label":"Time to first token","value":"57.66 ms","statistic":"p99, measured"},{"label":"Inter-token latency","value":"6.95 ms","statistic":"p50, measured"},{"label":"Output throughput","value":"1,085 tok/s","statistic":"measured"},{"label":"Capacity","value":"509 req/min","statistic":"measured"},{"label":"Peak VRAM","value":"14.38 GB","statistic":"measured under batch load"},{"label":"FP8 quantized request latency","value":"150.23 ms","statistic":"request latency, measured; percentile not recorded"}]},"headline":[{"value":"48.82 ms","label":"Time to first token","statistic":"p50, FP8 dynamic + batched serving"},{"value":"1,085 tok/s","label":"Output throughput","statistic":"measured, FP8 dynamic + batched serving"},{"value":"$0.151","label":"Per 1M output tokens","statistic":"GPU compute at measured saturation, FP8 dynamic + batched serving"},{"value":"0.982","label":"Quality vs FP16 baseline","statistic":"measured, FP8 dynamic + batched serving"}],"quality":{"score":0.9817,"evaluator":"first-token KL divergence (70%) + perplexity delta (30%) vs FP16 baseline","scaleNote":"1.0 = no detected degradation on this composite under the disclosed workload"},"limitations":["Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and not collapsed into a single reduction claim.","Cost per 1M output tokens is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees.","Receipts are added only when runs pass the same provenance gate."]}],"rentedServerlessListPrices":[{"provider":"Groq","item":"Llama 3.1 8B Instant","price":"$0.05 in / $0.08 out","unit":"per 1M tokens","sourceUrl":"https://groq.com/pricing","asOf":"2026-07-12"},{"provider":"DeepInfra","item":"Llama 3.1 8B Instruct Turbo","price":"$0.02 in / $0.03 out","unit":"per 1M tokens","sourceUrl":"https://deepinfra.com/pricing","asOf":"2026-07-12"},{"provider":"Groq","item":"Llama 3.3 70B Versatile","price":"$0.59 in / $0.79 out","unit":"per 1M tokens","sourceUrl":"https://groq.com/pricing","asOf":"2026-07-12"},{"provider":"DeepInfra","item":"Llama 3.3 70B Instruct Turbo","price":"$0.10 in / $0.32 out","unit":"per 1M tokens","sourceUrl":"https://deepinfra.com/pricing","asOf":"2026-07-12"},{"provider":"Together AI","item":"Llama 3.3 70B","price":"$1.04 in / $1.04 out","unit":"per 1M tokens","sourceUrl":"https://www.together.ai/pricing","asOf":"2026-07-12","note":"Llama 3.1 8B is no longer listed on Together's pricing page."}],"rentedDedicatedGpuListPrices":[{"provider":"Together AI","item":"H100 80GB, dedicated","price":"$5.49","unit":"per GPU-hour","sourceUrl":"https://www.together.ai/pricing","asOf":"2026-07-12"},{"provider":"Fireworks AI","item":"H100 80GB, on-demand","price":"$7.00","unit":"per GPU-hour","sourceUrl":"https://fireworks.ai/pricing","asOf":"2026-07-12"},{"provider":"DeepInfra","item":"H100 80GB","price":"$2.20","unit":"per GPU-hour","sourceUrl":"https://deepinfra.com/pricing","asOf":"2026-07-12"},{"provider":"Replicate","item":"H100","price":"$5.49","unit":"per GPU-hour","sourceUrl":"https://replicate.com/pricing","asOf":"2026-07-12"},{"provider":"AWS EC2","item":"p5.48xlarge (8x H100), us-east-1","price":"$55.04","unit":"per instance-hour","sourceUrl":"https://aws.amazon.com/ec2/pricing/on-demand/","asOf":"2026-07-12","note":"Whole-instance SKU (about $6.88 per H100-hour as context; single H100s are not sold separately)."},{"provider":"Google Cloud","item":"a3-highgpu-8g (8x H100), us-central1","price":"$88.49","unit":"per instance-hour","sourceUrl":"https://cloud.google.com/products/compute/pricing/accelerator-optimized","asOf":"2026-07-12","note":"Whole-instance SKU (about $11.06 per H100-hour as context)."}],"deploymentFreedom":[{"capability":"Own the optimized model artifacts","runinfra":"Yes, exported to you","serverlessApi":"No","dedicatedRental":"No, hosted on their infrastructure"},{"capability":"Deploy in your own cloud account","runinfra":"Yes, export kits","serverlessApi":"No","dedicatedRental":"No"},{"capability":"Deploy on your own GPUs","runinfra":"Yes","serverlessApi":"No","dedicatedRental":"No"},{"capability":"Run on a laptop (GGUF export)","runinfra":"Yes, where the model fits","serverlessApi":"No","dedicatedRental":"No"},{"capability":"Optimization measured on real GPUs before you pay for serving","runinfra":"Yes, receipts like the one above","serverlessApi":"Not applicable","dedicatedRental":"Not published"},{"capability":"Model keeps working if the provider delists it","runinfra":"Yes, you hold the artifacts","serverlessApi":"No, catalog changes retire models","dedicatedRental":"Provider dependent"},{"capability":"No coding required to optimize and deploy","runinfra":"Yes, agent-driven","serverlessApi":"API integration required","dedicatedRental":"API integration required"}],"notClaimed":["We have not measured competitor latency or throughput. Provider numbers on this page are their public list prices, nothing else.","Rented per-token prices and our measured cost per token are different economic units (pooled utilization vs a GPU you saturate), so we never chart them on one scale.","Per-token prices across vendors are not directly comparable: tokenizers differ, and Anthropic's own pricing documentation notes its newer models' tokenizer produces roughly 30% more tokens for the same text (platform.claude.com pricing docs, as of 2026-07-12).","Only modalities with measured, provenance-checked runs appear here. More receipts are added as runs pass the gate, not before."],"changelog":[{"date":"2026-07-12","change":"Added Llama 3.1 8B Instruct on NVIDIA A100-40GB, measured 2026-07-12 with a signed measurement proof."},{"date":"2026-07-12","change":"First public receipt: Qwen2.5 0.5B Instruct on NVIDIA T4, measured 2026-07-11. Provider list prices captured 2026-07-12."}]}