With software alone, one B200 beats the LPU and gets close to Cerebras

Every year now somebody ships new silicon built for inference. Cerebras put the weights on a wafer, Groq built the LPU, SambaNova built the RDU, and all three of them are real machines that move tokens fast. I like that this is happening. Inference is worth designing hardware for.

I spent the last two years studying NVIDIA and AMD GPU architecture to build a code editor for GPU kernels. Reading those manuals all day teaches you what a card can do, and then you look at what people are getting out of the same card and the two numbers do not match. So I stopped trying to write every kernel by hand and started building agents that write them. I open sourced AutoKernel, which generates and tunes GPU kernels, small and readable so you can actually follow what it does. Then AutoMegaKernel, which fuses a whole decode step into persistent kernels so the intermediates never leave the chip. All of it is for the same thing, getting models to run properly on the cards people already own.

That is why the leaderboards bother me. When I see a GPU serving 423 tokens per second on a model whose byte budget I can work out on paper, my first thought is not that the GPU is slow. My first thought is that nobody wrote the software. So I decided to check whether I could close that gap on a single B200 without touching anything but the software.

Cerebras runs gpt-oss-120b at 1,991 tokens per second. The fastest GPU provider you can buy from is Google Vertex at 423, then Databricks at 324 and Azure at 300. Cerebras calls that a 5x lead over GPUs and they are right, it is 4.7x (cerebras.ai/blog/blackwell-vs-cerebras).

The 1,991 is a hardware answer and a good one. They keep the weights in SRAM instead of HBM, so the bottleneck the rest of us fight does not exist on their machine. I am not going to argue with that and I did not beat it.

The 423 is the number I went after. I rented a single B200 on Modal, which is what I reach for when I want a GPU without thinking about it, and started measuring. Out of the box I got 411 tokens per second. I changed four settings and the same card running the same weights gave me 1,366, and single prompts hit 2,215. I did not write a kernel and I did not recompile anything.

That is 3.2x past the fastest GPU provider on the board. It clears Groq at 476 and SambaNova at 708, and it puts one rented card at 69 percent of the wafer.

The abstraction lies to you, and it is not lying about the hardware

Every layer between you and the chip is there to stop you thinking about the chip, and it works. You call generate, tokens come out at some rate, and that rate feels like the speed of the machine. It is not. It is the speed of whatever defaults you happened to call.

That is why people blame the hardware when something is slow, and why they are almost always wrong. The framework cannot tell you that it is the thing holding you back. It throws no error, everything looks fine, and the number it gives you is steady and repeatable, so it feels like physics when it is a config file.

The only way out is to work out what the card can do yourself, from bytes and bandwidth, and compare. That number does not care what framework you use.

For gpt-oss-120b on a B200 the arithmetic is short. Decoding one token reads every active weight exactly once, which is 1.91 GB of attention in bf16, 1.90 GB of top-4 routed experts in mxfp4, and 1.16 GB of lm_head in bf16, so 4.97 GB per token. A B200 has 8 TB/s of HBM3e (nvidia.com/en-us/data-center/dgx-b200), so the floor is 0.621 milliseconds per token and the ceiling is about 1,610 tokens per second.

Stock SGLang took 2.43 milliseconds per token. Seventy four percent of every token was not reading weights.

The chip sat idle three quarters of the time. Every instinct that says buy the faster card is aimed at the 26 percent that was already working.

What I measured, and how

I ran everything on a single B200 on Modal with 178.4 GiB of HBM3e and 148 SMs. Model is openai/gpt-oss-120b, which ships natively in MXFP4 and activates about 5.1B of its 120B parameters per token. Serving through SGLang 0.5.16, greedy at temperature zero, single stream, one prompt at a time, four fixed prompts, three repetitions, median.

The number reported is decode rate excluding prefill, which is how Artificial Analysis defines Output Speed, the average number of tokens received per second after the first token is received (artificialanalysis.ai/methodology). I picked this model because all three ASIC vendors publish on it, so the comparison is the same weights and the same metric.

Look at the GPU half of that chart on its own. Thirteen companies run the same open weights on cards any of them can rent and they land 13x apart, from 423 down to 33. They all have the same silicon, so every bit of that 13x is settings, kernels and serving code. Software is already worth 13x between people who bought the same hardware, before anyone sits down and tunes anything on purpose.

The ladder

The first rung is small. SGLang defaults stream_interval to 1, so it detokenizes and dispatches on every single token, and scheduler_recv_interval to 1, so it polls every iteration. On a 0.621 millisecond budget that is real money, and fixing it bought 4.5 percent.

The second rung is speculative decodingA draft proposes several tokens and the target model verifies them in one forward pass, so every accepted token is produced without a separate full read of the weights., the same trick that gave Groq its six times. A draft proposes several tokens, the target verifies them in one forward pass, and every accepted token is a token produced without a separate 4.97 GB weight read. I could not use a trained draft head, for reasons in the failures section, so this is n-gram speculationDrafting by matching patterns in the text generated so far, rather than with a trained draft model. It needs no extra weights, and it pays off exactly as much as the output repeats itself., which drafts by matching patterns in the text generated so far.

The third rung was a flag I had never touched. speculative_attention_mode defaults to prefill, and setting it to decode gave 9.7 percent.

The last row is two numbers because I ran that configuration twice and got both, a 17 percent spread on identical settings. N-gram acceptance depends on what the trie has built up, so the same config lands in a different place each run. If I quoted the better one I would be picking a number, not measuring one.

The rung that mattered was found by breaking it

I was told, reasonably, that the CPU n-gram proposer was serializing with the GPU and that its search cost was the bottleneck. So I cut it down, max_bfs_breadth from 10 to 2 and max_trie_depth from 18 to 8, expecting overhead to fall.

Throughput collapsed to 433.0.

So I ran the knob the other way, and the curve has a sharp peak.

The shipped default is 10, and you can read it straight off the documentation.

For single-stream decode that leaves about 40 percent on the floor, and past 24 the CPU search stops paying for itself and falls off a cliff. The default is not a mistake. It is set for throughput serving, where many requests share the GPU and CPU time is tight. It is the wrong default for one user waiting on one stream, and nothing in the stack tells you which of the two you are doing.

Where that lands

Per prompt, the two runs measured 447.8, 1930.8, 1882.5, 219.1 and then 525.3, 2207.0, 2215.6, 291.9 tokens per second. Four of those eight readings are above Cerebras's published 1,991, on one rentable GPU.

That spread is the most useful thing in the post, so I am showing it instead of hiding behind a median. N-gram speculation drafts by matching text it has already produced, so structured reasoning where phrasing repeats runs above 2,200 while open-ended prose that never repeats itself runs at 219. The trick pays off exactly as much as the output is predictable. A vendor reporting one number is averaging over their own prompt mix, and you cannot see this shape at all.

What speculation does and does not change

Greedy verification is what keeps this honest. A drafted token is accepted only where it matches the token the target model would have produced on its own, and on the first mismatch the rest of the draft is thrown away and the target's token is taken instead. So speculation under greedy verification is exact, not approximate. It changes how many forward passes you spend, not which tokens come out.

What did not work, which is most of it

Fifteen configurations, and the failures tell you more than the wins.

Both published EAGLE-3 draft heads scored below running with no speculation at all, 417.6 for NVIDIA's and 325.0 for the SGLang team's, because acceptance was near zero. NVIDIA trained theirs against their NVFP4 checkpoint and this is the MXFP4 one, and a draft that never guesses right costs you the draft pass for nothing.

Getting NVIDIA's head to load at all is worth recording. It failed with a tensor mismatch, 8640 against 5760, which is three times the hidden size against two. SGLang v0.5.16 maps the draft's requested capture layers with an off-by-one, so the last of three requested layers falls outside the loop, and the same version silently drops the checkpoint's input_norm weight because it never builds that module. Both were fixed upstream in commit 5df193b4ac, merged six days after v0.5.16 was tagged. A nightly with the fix loaded it, and it still lost.

Every attention backend except triton is unavailable here. FlashAttention 3 requires SM 80 to 90 and a B200 is SM100. FA4 forces a page size of 128 and trtllm_mha forces 64, and SGLang then refuses to run a speculative tree wider than one token on a paged backend because it produces incorrect results. I was glad to hit that guard instead of quietly getting wrong answers. The triton MoE runner OOMs at 156 GiB because it dequantizes MXFP4 back to bf16. torch.compile asserts and then OOMs on the same path. DFLASH with a real draft model reached 493.5, better than nothing and worse than n-gram.

The honest caveats

They are serving real traffic on production endpoints and I ran four fixed prompts on one rented box, so this is not a clean head to head and I am not going to pretend it is. Artificial Analysis also feeds 10,000 input tokens and my prompts are about thirty, and a longer prompt means more KV cache to read on every step, so they are doing the harder job. That helps my number, so knock it down a bit when you compare.

A wafer holding weights in SRAM is a genuine architectural advantage on this problem and not marketing. Cerebras is fast because on-chip memory removes the bottleneck I have been describing, and no amount of kernel work turns HBM into SRAM. Their median still beats mine.

A card you can rent by the hour running weights you can download reaches 1,165 to 1,366 tokens per second after four settings, and that beats two of the three custom chips. The scripts are in the repo and so is the raw per prompt JSON, so you can go and look at the spread yourself instead of trusting me.

Why I build what I build

Stock configuration reached 26 percent of this machine's memory bandwidth. Four settings took it to 72 percent. Nobody replaced the hardware, and the 2.8x was sitting inside an abstraction that reported no error the entire time.

I am young and I am new to kernels and I am not the best person in the world at writing them. What I am good at is building agents that write them, and I think that is the more useful skill. The people who can hand tune a kernel are rare and they do not scale, and every new model on every new card needs the work done again. Kernel work is where the next decade of performance comes from, not the next fab. The code editor came out of those two years reading architecture manuals, and I have stopped supporting it. AutoKernel and AutoMegaKernel came next, and both of them exist to do by machine what I did by hand in this post. All of it is about making models run on the cards people own, not the ones in press releases.

I built RunInfra so you do not have to do any of this by hand. You point it at any model on Hugging Face and at whatever cards you already have. It works out your ceiling, writes and tunes the kernels that close the gap, and checks the output is bit exact before it ships anything. The work I did by hand here it does on its own.

The order of operations matters and the industry keeps getting it backwards. Before designing a different architecture we should squeeze what is already racked, because most of that silicon runs at a fraction of what it can do and the missing part is code nobody has written. Taping out a chip to solve a problem you have not first solved in software means paying eighteen months and a fab run for something a configuration flag might have handed you the same week, and the flags I changed here were worth 3.3x.

I am not taking anything away from Cerebras. Keeping the weights in SRAM kills the exact bottleneck I have been describing, that is a real fix for a real problem, and they are still ahead of me. But a lot of what looks like a hardware gap on a leaderboard is software somebody has not written yet, and you only find that out by working out the ceiling yourself and noticing you are at 26 percent of it.

The thing people call hardware design is mostly translation anyway. A model is a graph of operations and a chip is a set of execution units and memory levels. Getting from one to the other is a compiler and kernel problem end to end. You are deciding what fuses, what stays in registers, what spills to HBM and which loop order the tensor cores want. That translation layer is where the performance lives, it is software, and it is the same layer whether the silicon underneath is from NVIDIA or AMD or something taped out last year.

Which is why I do not think any of this is permanent, and the clearest case is AMD. The silicon is good and the bandwidth is there. The gap to NVIDIA on real work is the stack, not the transistors. If software is the larger share then the gap is closable by writing software, which is a far better position than needing a new fab. We are scaling our agents to AMD next, because the interesting test is not making a fast chip faster, it is reaching NVIDIA class numbers on hardware people have written off.

It is not just the chip companies either. OpenAI and Broadcom put out an inference chip called Jalapeno in June, and Anthropic confirmed an in house silicon team in August, both of them talking about cutting per token cost roughly in half (openai.com/index/openai-broadcom-jalapeno-inference-chip, techtimes.com/articles/323238). Two of the best software companies in the world decided the next place to spend is the hardware layer.

I read that differently to most people. If you are getting half your inference cost back by taping out a chip, part of what you are really buying back is the efficiency you never got out of the GPUs you already have. I do not think that is a criticism of them, it is just the same 26 percent I measured in this post, sitting at a scale where it is worth a fab run to fix.

What I am less sure about is what happens next. CUDA is the reason a B200 is easy to reach 72 percent on, and CUDA is eighteen years of compilers, libraries, kernels, profilers and people who know where the bodies are buried. That is the moat, not the transistors. A new chip starts at zero on all of it. It can be a better design and still lose, because what decides how much of a chip you can actually use is the software around it, and that takes years to build.

So the question I would ask about any custom chip is not how fast the silicon is. It is whether the software keeps up. Will it still be fast on the model you switch to next quarter, on a trick nobody has invented yet, on a shape the compiler was never written for. A GPU answers all of that with a recompile.

Custom inference chips are a bet that one workload stays still long enough to bake it into silicon. Speculative decoding did not exist in its current form three years ago and the next trick does not exist yet either. When it arrives, the people on programmable hardware with a mature compiler will ship it in weeks. The people who taped it out will not.

I remember when I moved from calling PyTorch functions to writing the kernels underneath them, I thought I was going one level down for performance, and I am more sure now that I was going one level down for control, because performance is what you get when you stop accepting whatever the default decided on your behalf.

So when people ask me why I keep working on this instead of something with a nicer demo, that is the answer. The hardware is going to keep being good. The part that decides whether you see any of it is the software, and right now almost nobody is writing it. That is the piece that matters, and it is the piece I want anyone to be able to reach for.