$0.09 and $290.12: What Actually Moves Your Inference Bill

Nine cents. Two hundred and ninety dollars and twelve cents.

Those are both the price of one million output tokens. Same unit, same dataset, read off vendor pages in the same week. The first is gpt-oss-120b on a single MI355X. The second is Qwen3.5 on eight H100s.

Neither is a typo and neither is a trick. And almost none of the gap between them is the model.

So I went looking for what it actually is. Here is what I found, measured across 24 providers and 378 GPU rental rates, ranked smallest to largest:

Five levers. The two everyone talks about are the two smallest. This post walks up the list.

First, the question everyone actually asks

Is self-hostingRunning model inference on compute you rent or own instead of buying each request from a hosted model API. cheaper than the API?

Here is the like-for-like comparison. Same weights, same model id, cheapest published option on each side.

At 90% utilizationThe fraction of paid accelerator time during which the hardware is doing useful inference work. self-hosting takes two of three. At 30%, which is generous for most real deployments, the API takes two of three.

So the answer to the question is it depends on your utilization, which is unsatisfying and also true. But the more useful finding is that every number in that table is downstream of decisions that are not in the table.

Lever 1: the accelerator. 2.3x.

Every self-hosted win above is on AMD MI355X. Not one is on NVIDIA.

MI355X on Vultr is $2.59 per GPU-hour. The cheapest B200 rate we found anywhere is $5.89. Under half the price, against higher measured throughput on DeepSeek-R1.

This is not AMD is better. It is that the accelerator market is priced far less efficiently than the model market, and almost nobody shops it.

The evidence for that: none of the eight mainstream GPU providers we surveyed first published an on-demand per-GPU MI355X price at all. The cheapest capacity on the market is also the hardest to get a number for. Every row in our inputs we could not price is left blank rather than costed against a guess.

Lever 2: which provider runs the same weights. 3.2x.

Identical model id. Identical weights. Standard tier. One API call away from each other.

gpt-oss-120b        ##################################  3.0x  $0.25 -> $0.75
gemma-4-31b-it      #############################       2.6x  $0.38 -> $0.97
qwen3.7-max         #######################             2.0x  $3.75 -> $7.50
gpt-oss-20b         #######################             2.0x  $0.15 -> $0.30
gemma-3-27b-it      #####################               1.9x  $0.16 -> $0.30
glm-5.2             #####################               1.8x  $2.40 -> $4.40
kimi-k2.6           ###############                     1.3x  $3.40 -> $4.50

This is the cheapest saving available to anyone reading this and it requires changing a base URL.

The spread is not uniform, which is the part worth knowing. Kimi K3 is $3 in and $15 out at seven hosts including Moonshot's own API, to the cent. There is no arbitrage there. Llama-3.3-70B ranges 3.2x. Check per model, because the intuition transfers badly.

Lever 3: your batching config. 3.5x.

Same model. Same 2x H100. Same rental rate. Only concurrencyThe number of requests a serving system processes at the same time. changes.

   658 tok/s  ##################################  $5.60
   998 tok/s  #######################             $3.69
 1,385 tok/s  ################                    $2.66
 1,825 tok/s  ############                        $2.02
 2,282 tok/s  ##########                          $1.61

A config flag moves cost 3.5x. That is more than the gap between the cheapest and most expensive GPU on the market.

It is a real tradeoff, since higher concurrency means worse per-user latency. But it is a tradeoff you can tune in an afternoon. The hardware purchase is not.

People agonise over which accelerator to buy and then run it at concurrency 8.

Lever 4: utilization. 9x.

Same hardware, same rental rate, same config. Only the fraction of the hour the box is actually working changes.

Nine times, from an assumption.

Utilization is the input everyone estimates for themselves and nobody measures first. It is also where the honest version of the self-hosting question lives. If you can name your real sustained utilization, you already know your answer.

The wall underneath it

There is a structural fact hiding under utilization that changes how you should think about the whole problem.

Self-hosting is a fixed monthly bill. An API is purely variable. So the intuition is that self-hosting always wins eventually, once volume is high enough. Just wait for the lines to cross.

The lines do not always cross, because the box has a ceiling.

Take Qwen3.5 on 8x H100:

fixed cost                     $11,462 / mo
capacity at 30% utilization        367M tokens / mo
break-even vs Sonnet 5 at $10    1,146M tokens / mo

Break-even sits at three times what the hardware can physically emit. There is no volume at which this configuration beats that API price. Not at ten times the traffic, not at a hundred.

When capacity is below break-even, more demand does not help you. It means you needed a different box, or no box.

This is the common case against cheap models, and it is why self-host to save money fails most often on exactly the workloads people try it on first.

Lever 5: reasoning tokens. 14.3x.

On reasoning models, the tokens you are billed for are not the tokens you see. Reasoning tokensOutput-rate tokens used by a reasoning model before it returns the visible answer. bill at the output rate and arrive invisible or summarized.

A request to gpt-5.6-sol at $30 per 1M output that returns 300 visible tokens after 4,000 reasoning tokens bills 4,300 tokens, not 300.

4,300 / 1,000,000 * $30 = $0.129
  300 / 1,000,000 * $30 = $0.009

14.3x, from a number that does not appear in your response body.

That gap is larger than every price difference between every vendor in this entire dataset, combined with every hardware decision above it. If you are comparing providers per token while your reasoning effort is set to high, you are optimizing the smallest term in the equation.

The token counts in that example are illustrative. The $30 rate is verified. Read usage.completion_tokens_details.reasoning_tokens on your own traffic rather than trusting anyone's ratio, ours included.

Two traps that cost real money

What is not in the dataset, and why that matters

Kimi K3 and GLM 5.2 are absent from every self-hosting table here.

Both are priced by eight or more hosts. Both are in the pricing data. Neither has a public throughput datapoint that states hardware, engine version, precision, concurrency, and sequence lengths together. Without all five you cannot compute a serving cost without guessing.

So there is no row.

The blank is the finding.

Two of the most widely served open models on the market have no reproducible published serving benchmark. That is not a gap in our data.

Where this is wrong

Not decoration. These are the weaknesses we know about.

The formula charges the entire GPU bill to output tokens, which overstates output cost for input-heavy workloads. Utilization is assumed, not measured, and real traffic is diurnal. The throughput points are somebody else's operating point, mostly published by parties with an interest in the result, at favorable sequence lengths. Quantization is not held constant. On-demand is the worst GPU price and committed contracts would improve every self-hosted row. Break-even uses uncached output price, so agent workloads with stable prefixes swing further toward the API than shown.

Engineer time is excluded entirely. One engineer at a loaded $200k a year is about $16.7k a month, which buys a great many tokens. Every self-hosting number here gets worse when you add them back.

When the API is the right choice

Most of the time.

Under roughly 100M output tokens a month against a cheap model, the GPU bill alone exceeds the API bill before you hire anyone. If your traffic is bursty you pay for idle GPUs, and utilization is the dominant term. If you have a tight p99, note that the cheap end of every self-hosted row above assumes a saturated box, which is the same thing as a queue. The configuration that makes the spreadsheet work is the one that makes the service worse.

And if you need frontier capability, no open model reaches that tier, so the question does not arise.

Self-hosting wins on things that are not price: data residency, no rate limits, model pinning, custom weights, no vendor deprecation schedule. If you are self-hosting, those are usually the honest reasons.

The thing to take away

Three of the five levers are software. The concurrency config, the utilization you actually sustain, and the reasoning tokens you emit are all decisions in your serving stack, and together they span 9x, 3.5x, and 14.3x. The two hardware and vendor decisions everyone argues about span 2.3x and 3.2x.

That ordering is why we work on kernels rather than on procurement. RunInfraThe company publishing this article, which makes models run faster on GPUs its customers control. exists to move the same box further up that curve, because the levers with the largest range are the ones inside your serving stack, not the ones on a rental price list.

Nothing about that changes the conclusion at the top of this post. At 30% utilization the API usually wins, and we would rather publish that than pretend otherwise.

What we sell, so you can weigh the bias

Two things, both concrete.

Optimized serving packages for specific models. The current flagship is Kimi K3. We ship the whole recipe: tuned kernels and the serving config. The package contains zero weight bytes. It pins the weights to a specific Hugging FaceThe upstream source from which the buyer's node pulls the pinned Kimi K3 weights. revision, and your own node pulls them directly from Hugging Face. We do not store, mirror, re-upload, or redistribute the weights. You upload the recipe to whatever GPU provider you already use and run it on your own compute. No endpoint of ours in the path, no per-token cut.

8x B300, single node
vLLM, TP8 + EP, 40960 context
weights pinned to a specific HF revision
built entirely on public dependencies, so reproduce.sh actually runs

$2,300 one-time + $200/mo for continuous optimization

The public-dependency constraint is deliberate and it cost us time. Serving K3 the fast way today routes through private forks, so the config most people quote is one you cannot reproduce. Ours is the config you can.

The measured 2.1x result below compares K3 Turbo with the same configuration with speculative decoding turned off at concurrency 1; median latency p50 was 18,730 ms at baseline and 8,907 ms optimized, and the advantage narrows as concurrency rises, having inverted to minus 9.46 percent at concurrency 32 in the prior measurement window; this was measured with an unshipped runtime.

Custom optimization for anything else. Any open-weight model, on your hardware or your cloud, optimized and deployed. It starts as a conversation at runinfra.ai. Tell it what you are serving and on what, and we scope it from there.

Everything in this post argues that the software layer is where the money is. That is our thesis and our business, which is exactly why we published the data that undercuts it.

Every number here is in the repo, with a source URL and the date it was read. Dated snapshots are immutable. If a number is wrong, open an issue with the source and it gets fixed.

Data is CC BY 4.0A Creative Commons license that permits reuse with attribution.. Reuse it, link back.