Speculative decoding inverts with concurrency, and the draft horizon is the knob

At concurrency 1, turning DSpark speculative decoding on for Kimi K3 doubled the rate a single user feels: 55.28 to 118.22 tok/s p50, plus 113.84 percent, a 2.14x ratio (results/raw/dspark-comparison.md, arms results/raw/dspark-armA-single-stream.json and results/raw/dspark-armB-single-stream.json). At concurrency 32, in the same container, on the same eight B300s, the same change measured MINUS 9.46 percent aggregate throughput (301.28 to 272.77 tok/s) and dragged time to first token from 7.4 seconds to 114.5 seconds (7398.63 to 114549.13 ms p50, results/raw/dspark-armA-sweep-A.json and results/raw/dspark-armB-sweep-A.json).

One variable changed between those two columns. Both arms carried identical flags, both ran sequentially inside one container on one wheel install, and the comparison tool verified 26 request-profile fields identical across arms, including a sha256 fingerprint of the prompt set at every concurrency point, before it rendered a single number.

That sign flip is the finding. What follows is why it happens, what retired it, and how to find the same knee on your own stack.

The mechanism, stated plainly

Drafting spends compute to buy latency. A speculative step runs a small draft model forward k times, then runs the full target model once over all k+1 positions and keeps the accepted prefix. The arithmetic is worth it exactly when the target model's step is not compute bound, which at batch 1 on a 2.8T mixture-of-experts model it emphatically is not: verifying five positions costs almost the same wall time as verifying one, because both are dominated by streaming expert weights out of HBM.

Raise concurrency and that free space disappears. The draft passes now compete with real prefill and real decode for the same SMs, and the extra verify positions stop being free because the batch already fills the machine. Two secondary effects in the raw files confirm the picture rather than contradicting it. TPOT is worse in every speculative arm even where throughput is better (18.09 to 24.94 ms p50 at c=1 in the k=7 window), which is what you expect when each step does more work and emits several accepted tokens at once. And at c=32 the end-to-end p50 barely moved (215245.05 to 212831.87 ms, plus 1.12 percent): in a closed loop the damage lands almost entirely on time to first token, because prefill is now queued behind draft work.

The acceptance decay is where the waste was

The Prometheus counters from the k=7 window are the evidence that the tail of the draft was pure cost. results/raw/twocol-armB-metrics-after-t30.txt records 60944 drafts, 426608 draft tokens (exactly 7.0 per draft) and 126490 accepted, with per-position acceptance of 44225, 29747, 19984, 13513, 9086, 6095 and 3840. As a share of drafts that is 72.57, 48.81, 32.79, 22.17, 14.91, 10.00 and 6.30 percent. Positions 5, 6 and 7 were drafted and verified on every step and accepted less than one time in six.

A later, cleaner instrument measured the same shape at k=4 on the shipped configuration. results/raw/accept-acceptance-probe.json records 3478 drafts, 13912 draft tokens and 5738 accepted at temperature 0.0: 2.6498 accepted tokens per verify step, a 41.24 percent draft acceptance rate, and per-position acceptance of 0.721, 0.462, 0.297 and 0.170. The same probe at temperature 0.8 measured 2.6028 per step and 0.705, 0.447, 0.279, 0.172, so this decay is not an artifact of greedy sampling.

A curve that falls from 0.72 to 0.17 over four positions tells you where its own knee is. Each additional draft position costs a fixed amount of draft compute and a marginal amount of verify compute, and returns the product of all the acceptance rates before it. Past the point where that product drops below the cost, extending k is a tax you pay on every step.

The one-variable window that retired the inversion

The counters said three of seven drafted tokens were being paid for and thrown away, so the next window changed exactly one thing: num_speculative_tokens 7 against 4, everything else the shipped configuration, comparison exit 0 (results/raw/speck7v4-comparison.md).

k=4 won at every point measured. Single-stream decode p50 114.00 to 115.65 tok/s, plus 1.45 percent. Sweep c=1 plus 7.14 percent per GPU. c=8 plus 15.55 percent with TTFT p50 falling 10760.66 to 6111.00 ms. c=32 plus 28.16 percent, 273.84 to 350.96 aggregate tok/s, with TTFT p50 falling 118123.33 to 28160.27 ms.

The headline window was then re-run at k=4 against speculation off (results/raw/k4pub-comparison.md). Single stream 55.15 to 119.84 tok/s, plus 117.30 percent, so essentially the whole per-user win survives the shorter horizon. At c=8, plus 17.83 percent with TTFT p50 improving 12277.83 to 7008.16 ms. At c=32, throughput is MINUS 0.83 percent, which sits inside the measured throughput noise floor of about 0.4 percent and should be read as flat, while TTFT p50 improves 35456.73 to 29434.98 ms.

So the inversion is not a law about speculative decoding. It was a property of one draft horizon on one drafter, and it went away when the horizon matched the drafter's acceptance curve.

The other side of the knee

k=3 was measured too, one variable against the shipped k=4 (results/raw/speck4v3-comparison.md). It loses the metric the product leads with, single stream 118.27 to 113.85 tok/s, minus 3.74 percent, and buys a few percent of batch throughput plus real latency: c=32 plus 4.70 percent with TTFT p50 28005.49 to 17419.78 ms, plus 37.80 percent.

k=7 lost everywhere to k=4. k=3 loses the per-user headline and buys latency under load. That is a knee, and k=4 is on it for a per-user-speed target.

The deployment rule

Two rules, and the first one was retired by measurement rather than defended.

Version 3 of this package shipped k=7 and told buyers to switch speculation off for batch work, because at c=32 it measured minus 9.46 percent with a 114.5 second TTFT. That rule is now retired: at k=4, c=32 throughput is flat inside the noise floor and time to first token is better with speculation on.

The rule that replaces it: run speculation on across the whole curve, and set k from your own acceptance measurement rather than from a default. Expect the per-user gain to be large at low concurrency (plus 117 percent measured at c=1), to narrow with load (plus 17.83 percent at c=8), and to reach approximately nothing at c=32 while latency still improves. A batch-first deployment may prefer k=3 for its measured latency win, at a measured 3.74 percent cost to single-stream speed.

How to find your own knee, at the cost of two windows

1. Serve your real configuration once and scrape vllm:spec_decode_num_drafts_total, vllm:spec_decode_num_draft_tokens_total, vllm:spec_decode_num_accepted_tokens_total and the per-position acceptance series, with snapshots bracketing the traffic. Snapshots taken before traffic are all zeros and can source nothing; that mistake is on the record in this repository.
2. Compute accepted tokens per step and per-position acceptance. Read where the per-position curve crosses under roughly 0.2. In our measurements that crossing sits between position 4 and position 5, and the measured k knee landed exactly there.
3. Run one two-arm window changing only k, with both arms in one container on one wheel install, and require the request profile to be verified field by field before any number renders.
4. Model the ceiling before buying more windows. Per-user decode rate equals accepted tokens per step divided by step time. Measured here: 23.38 ms step, 2.6498 tokens per step. That makes the k=4 ceiling at perfect acceptance 213.8 tok/s, which is how this project concluded that no acceptance tuning at k=4 reaches 200 and that a better drafter, not a longer horizon, is the lever.

What these numbers do not say

The decode rate here is defined as (output tokens minus 1) divided by (last chunk time minus first chunk time), TTFT excluded, p50 over eight timed single-stream requests at a 1024 prompt and 1024 output profile. With speculation on, one stream chunk can carry several tokens, so this is reported tokens over the chunk-time span and must never be described as chunks per second (the committed raw file for that window carries that caveat inside the file).

Cross-window comparisons are indicative only. The same k=4 configuration measured 118.27 tok/s in one window and 119.84 in another, a 1.3 percent spread that is wider than the 0.4 percent throughput noise floor measured on the sweep metric.

One structural confound is worth naming because it cuts against the speculative arm being a clean isolate: on this engine a speculation-enabled server does not construct the in-tree fused non-speculative KDA decode kernel at all, so a speculation-on versus speculation-off delta conflates the drafter with the loss of that kernel. The k versus k windows do not have this problem, which is one more reason the k=7 against k=4 pair is the cleaner result.

No accuracy gate ran inside these windows. Speculative decoding with a correct verifier is distribution preserving by construction, which is why this was treated as a configuration change rather than a numerics change, and it is stated here rather than implied.