Quantization became the default way to speed up inference because it is the easiest one. In vLLM it is one flag or one swapped checkpoint (docs.vllm.ai), while doing the real optimization work like writing better kernels takes months. So people swap the checkpoint and move on, and the same people who would never ship an unreviewed prompt change end up shipping a quantized model without running a single eval, because the flag makes it feel like an infra setting when it is actually a model change
Quantization is a trade, not an optimization. You save memory and you pay for it in quality, and the cost is hard to see because post training quantization moves every weight a little while the standard benchmarks barely react, since they are saturated and they test one step at a time. The real loss shows up in long reasoning chains and agentic coding runs where hundreds of dependent decisions stack on each other, so the benchmark tells you the two models are the same while your agent's success rate tells you they are not. This post is about the serving stack that gets the speed without paying that cost
The compounding math
If quantization noise flips one decision per step with probability p, then over n dependent steps the task survives with probability (1-p)^n. A 2 percent hit per step sounds free until you run 20 steps, because 0.98^20 = 0.67 and a third of your tasks now fail, and at 50 steps you are down to 0.36
If you hold p at half a percent instead, survival over 20 steps stays at 0.90, and that is the difference between an agent you trust and an agent you babysit. The model never got visibly dumber, it just got 2 percent noisier per decision and the compounding did the rest
What lossless means
Lossless inference means the stack gets faster while the model stays exactly the same model, and I split it into two tiers
Bit exact means the optimized stack returns the same logits as the reference on every input, f_opt(x) = f_ref(x). Kernels and fusion, compilation, KV reuse, scheduling and lossless weight compression all live in this tier because they change how the math runs without changing the math itself
Distribution exact means the optimized stack samples from the same distribution as the reference, P_opt(y | x) = P_ref(y | x). Speculative decoding with rejection sampling lives here, where single samples can differ but the distribution never does
Everything else, PTQ, pruning and distillation, changes the function itself, which means you are serving a different model under the same name. This definition matters because it moves the burden of proof, since a lossless change needs no eval run when the outputs are the evidence, while a lossy change needs a full eval on your workload and not just on MMLU. Bit exact also composes, because a pipeline of bit exact stages is still bit exact, so I can stack five of them without running five eval cycles
The lossless stack, bottom up
The stack starts with kernels. Inference at serving batch sizes is memory bandwidth bound, since an H100 SXM moves about 3.35 TB/s from HBM while doing about 989 dense BF16 TFLOP/s (nvidia.com/en-us/data-center/h100/), so a kernel earns its speed by moving fewer bytes rather than by doing less math. FlashAttention proved the point years ago by computing exact attention with reordered memory traffic and getting big speedups with identical outputs (arxiv.org/abs/2205.14135), and the reordering is valid because online softmax computes the same normalization from running statistics, so nothing about the result changes
Fusion pushes the same idea further by keeping intermediate values in registers and shared memory instead of round tripping through HBM, and a megakernel takes it to the limit by running the whole decode step as one launch. I built this myself. On the same H100 my engine bonsai-turbo runs 1.76x the vendor's own llama.cpp fork, going from 85.5 to 151.1 tokens per second with the same outputs (github.com/RightNow-AI/bonsai-turbo), and my kernel search system AutoKernel reached 1.31x over cuBLAS on its best kernels across 95 experiments spanning 18 to 187 TFLOPS while holding first place on the B200 vectorsum leaderboard (github.com/RightNow-AI/autokernel)
Speculative decoding comes next. A small draft model proposes g tokens, the target model verifies them in one pass, and a modified rejection sampling step keeps the output distribution exactly the target's, which was proved independently twice in 2023 (arxiv.org/abs/2211.17192, arxiv.org/abs/2302.01318). The expected number of tokens per target pass is (1-a^(g+1))/(1-a) at acceptance rate a, so draft accuracy is everything, and the EAGLE line is what drove it up (arxiv.org/abs/2401.15077). EAGLE-2 states it plainly ("the distribution of the generated text remains unchanged") and reports 3.05x to 4.26x (arxiv.org/abs/2406.16858), while EAGLE-3 reports up to 6.5x (arxiv.org/abs/2503.01840)
Then there is lossless weight compression. BF16 weights carry less than 16 bits of real information per parameter, so DFloat11 entropy codes them down to about 11 bits, which lands at about 70 percent of the original size with outputs that stay bit for bit identical to the original model (arxiv.org/abs/2504.11651). That covers most of the memory saving that quantization promises without the quality bill
KV reuse is the next layer. Paged attention ended KV cache fragmentation and raised real batch sizes (arxiv.org/abs/2309.06180), and prefix caching with radix trees makes shared history get computed once (arxiv.org/abs/2312.07104). This is safe because cache entries are a pure function of the prefix, so reusing them at the same positions cannot move the logits
Scheduling sits on top. Continuous batching refills the batch at token boundaries instead of waiting for the longest request to finish (usenix.org/conference/osdi22/presentation/yu), chunked prefill stops a long prompt from stalling everyone else's decode (arxiv.org/abs/2308.16369), and prefill decode disaggregation puts the compute bound half and the bandwidth bound half of the workload on different machines (arxiv.org/abs/2401.09670). None of these touch a single logit
The moving line
The lossless boundary is drawn at the shipped checkpoint, and the checkpoint itself is moving. Kimi K2 Thinking shipped with native INT4 weights because Moonshot ran quantization aware training in post training, put INT4 on the MoE weights and reported every official benchmark at INT4 with roughly 2x faster generation (huggingface.co/moonshotai/Kimi-K2-Thinking). OpenAI shipped gpt-oss with MXFP4 MoE weights at 4.25 bits per parameter, which covers over 90 percent of the parameters and is the reason gpt-oss-120b fits on one 80 GB GPU (arxiv.org/abs/2508.10925)
For these models serving INT4 or MXFP4 is not lossy quantization, it is the checkpoint. The reference you must not degrade is already low precision, and the vendor already paid the quality cost during training where it belongs, so quantization is turning into a training decision while squeezing a BF16 checkpoint after the fact stays a serving hack. My test is simple, if the weights you serve are the weights the vendor benchmarked then you are serving the model, and if you squeezed them afterward then you are serving your own edit of the model
Verification
Lossless is a claim you can test, and I test it with logit parity. You run the same prompts through the reference stack and through yours with greedy decoding, then compare the maximum absolute logit difference per token, where zero is the pass bar for a bit exact claim and a distribution test is the bar for speculative decoding. I published this for bonsai-turbo, where 32 of 32 prompts pass against the vendor fork across all four engine configurations (github.com/RightNow-AI/bonsai-turbo). When parity fails the diff shows me where to look, and it is usually a fused kernel that cut a corner, a RoPE mismatch or a sampler that renormalizes differently
Moonshot now runs the same kind of check on everyone who serves its model. The K2 Vendor Verifier sends 4,000 identical requests to every third party K2 provider and scores them against the official API on tool call trigger F1 and schema accuracy (github.com/MoonshotAI/K2-Vendor-Verifier), and the spread was real, with schema accuracy running from 100 percent at the best providers down to about 73 to 76 percent on stock open source engines before fixes landed. Model quality became a serving stack property and model vendors have started measuring it
Floating point addition is not associative, so the reduction order inside a kernel changes the low bits, and that order changes with batch size. Thinking Machines measured the consequence when 1,000 temperature zero completions on stock vLLM gave 80 distinct outputs, while batch invariant kernels brought all 1,000 back bitwise identical at 1.6x to 2x the runtime (thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), and Chen et al. scoped their speculative decoding proof "within hardware numerics" for the same reason (arxiv.org/abs/2302.01318). So lossless means the stack adds no error beyond the numerics you already accepted, and it does not mean bitwise identical outputs across batch sizes unless you also pay for batch invariance
Why lossless wins
The economics settle the argument. Quantization saves a fixed factor once and keeps paying quality on every step of every task, while lossless wins multiply with each other, kernels times speculative decoding times KV reuse times scheduling, with every term keeping the model exact. One term alone already shows what lossless buys on real hardware:
I remember when I started working on this, I thought lossless would be the future, and I am more sure of it now. At RunInfra I build exactly this, serving open models on the full lossless stack and verifying it with logit parity against the reference
Written by
