Benchmarks methodology

The public benchmarks page at runinfra.ai/benchmarks publishes measured results from real optimization runs on real GPUs. This page documents how those numbers are produced, the rules that decide what ships, and what the page deliberately does not claim.

​

What a benchmark receipt is

A receipt is the publication unit of the benchmarks page. Each receipt covers one model on one GPU and carries everything needed to read its numbers honestly:

• The model, the hardware, and the measurement date.
• The optimization method, stated per configuration. For example: continuous-batching serving tuning plus FP8 dynamic quantization as the promoted winner, measured per configuration.
• The workload: RunInfra’s automated load test, run identically on the baseline side and the optimized side of the same GPU.
• The GPU cost basis recorded with the run, including the hourly rate used.
• Bar charts with one unit per chart and a stated direction (lower is better or higher is better). The statistic identity is printed on every row, for example “p50, measured”.
• A baseline fact table (FP16 serving) and an optimized fact table. Every metric carries its spelled-out statistic next to the value.
• Four headline stats. Each one shows its statistic plus configuration identity everywhere it renders, on the page and on the share image.
• A quality score with its evaluator and scale note printed alongside.
• The receipt’s own limitations, published with the numbers they qualify.

​

The provenance rule

Measured numbers come from real optimization runs whose results are stored with per-metric provenance flags. A number ships on the benchmarks page only when:

1. Its provenance flag says measured.
2. Its workload is reconstructable.
3. Its statistic is named next to the value.

Three labeling rules follow from this:

• Configurations are never mixed into one unlabeled result. A serving-tuned FP16 configuration and a promoted FP8 winner are different configurations. Every number is labeled with the configuration that produced it.
• Differently defined statistics are never collapsed into a single reduction claim. Baseline latency is a request-latency percentile; optimized serving latency is time-to-first-token. They are labeled separately and never combined into one headline reduction.
• Charts hold one statistic family on one scale. A p50 vs p99 chart plots the same statistic from the same configuration. A baseline number defined differently lives in the fact table, never on that scale.

When a detail was not captured, the label says so. For example, a metric can carry the statistic “request latency, measured; percentile not recorded” rather than implying a percentile that was never taken.

​

The quality score

Quality is scored against the FP16 baseline with a composite evaluator: first-token KL divergence (weight 0.7) plus perplexity delta (weight 0.3). The scale note ships with every score:

1.0 = no detected degradation on this composite under the disclosed workload

The score, the evaluator definition, and the scale note print together on the receipt. A quality number never appears without them.

​

The cost basis

Cost per 1M output tokens is computed by the measurement engine from measured token throughput and the GPU rate recorded with the run. It is a GPU compute estimate at measured saturation. It excludes idle time, CPU, memory, networking, redundancy, and platform fees, and every receipt says so. Each receipt also states its own GPU basis, for example A100-40GB compute at the rate recorded with the run ($2.10/hr), or T4 compute at the serverless rate recorded with the run ($0.59/hr).

​

Why rented list prices never share a chart scale with measured numbers

The benchmarks page shows what renting the same class of open-weight model costs, using public list prices from provider pricing pages, each cited with a source URL and an as-of date. Those prices live in their own tables with their own units and are never merged into one chart scale with RunInfra’s measured numbers, because they are different kinds of numbers:

• Different economic units. A pooled per-token rate and a GPU you saturate yourself have different utilization models. Charting them on one scale would imply a comparison the data does not support.
• Different tokenizers. Per-token prices across vendors are not directly comparable because tokenizers differ in how many tokens they produce for the same text.
• Different statistics. Provider list prices are not measurements; they carry no workload, percentile, or configuration identity.

Input and output token rates stay separate in the price tables; they are never averaged or blended. Whole-instance cloud SKUs (for example 8x H100 instances on AWS or Google Cloud) are shown as whole-instance prices, with any per-GPU figure marked as context only.

​

The machine-readable dataset

Everything the page publishes is available as JSON at runinfra.ai/benchmarks/dataset.json. The dataset is serialized from the same module the page renders, so it cannot drift from what visitors see, and it passes the same publication gate (no internal run identifiers, cited prices only).

​

What is not claimed

The page renders these statements verbatim, and they apply to this documentation too:

• We have not measured competitor latency or throughput. Provider numbers on this page are their public list prices, nothing else.
• Rented per-token prices and our measured cost per token are different economic units (pooled utilization vs a GPU you saturate), so we never chart them on one scale.
• Per-token prices across vendors are not directly comparable: tokenizers differ, and Anthropic’s own pricing documentation notes its newer models’ tokenizer produces roughly 30% more tokens for the same text (platform.claude.com pricing docs, as of 2026-07-12).
• Only modalities with measured, provenance-checked runs appear here. More receipts are added as runs pass the gate, not before.

​

How receipts get added

Receipts are added only when runs pass the same provenance gate, never before. Every addition is recorded in the changelog on the page and in the dataset, with the measurement date and the date provider prices were captured. If RunInfra ever measures competitor latency or throughput, it will be with a disclosed harness, like everything else on the page.

​

Next steps