Day zero on a 2.8T model: five walls between released weights and a served endpoint

Kimi K3's weights went public on 2026-07-27. Engine support merged to vLLM main on 2026-07-29, and the same day's nightly wheel, vllm 0.26.0, was inspected and contains zero kimi_k3 files. Merged to main is not installable. For a week, serving this model meant building from source and walking into five distinct failure walls, and the failure at the top of the pile was not a performance problem. It was an engine that would not start.

Here is the proof, from a two-arm window run on 2026-07-30 with both arms in one container on eight B300s. Arm A was stock upstream defaults. It failed to serve. results/raw/twocol-armA-vllm.log line 448 reads:

AttributeError: type object 'ActivationType' has no attribute 'Situ'. Did you mean: 'Silu'?

The failing chain is verbatim in the same file: the call at 443, the resolution attempt at 446, the worker RuntimeError at 605, engine core init failure at 714. Arm A produced no benchmark artifact of any kind, which is why this repository publishes no stock-versus-tuned delta for that window and never will. It is a measured serves-at-all result, not a marketing claim.

Three of the five walls are worth another team's time. Two you hit once and move past.

Wall 1: a three-day-old race in a third-party artifact bucket

Kimi K3's mixture-of-experts layers use a gated activation the model authors call SiTU. The public FlashInfer release the ecosystem was pinning to did not have it, and the interesting part is that adding it required three layers to line up, only one of which is Python.

The Python enum is the visible layer. At release v0.6.16rc3 (2026-07-28) ActivationType ends Identity = 9, InvalidType = 10. At v0.6.16rc5, published 2026-07-30T07:10:13Z, it reads Situ = 10, InvalidType = 11. That is the layer the traceback complains about, and it is the layer a frustrated engineer is most tempted to hand-patch.

Do not hand-patch it. The load-bearing layer is the third one: the prebuilt TRT-LLM-Gen device cubins, which are not compiled by nvcc from the FlashInfer tree at all. They are fetched from an NVIDIA artifact bucket whose path is a string constant in the package. We fetched the generated metadata header from both bundles directly, no authentication required, and counted occurrences of the kernel-name token siTuGlu:

Full derivation and URLs in docs/SITU-GAP.md section 3. Of those 292, 110 are the MXFP4-weights by MXFP8-activations shape this model's expert path wants, and all 292 carry the sm100f arch tag. A representative kernel name ends ..._clmp_siTuGlu_dynB_sm100f.

That is the whole lesson, and it generalizes: the activation is a token in the compiled kernel identity, not a runtime flag. SiTU is a distinct compiled kernel family, not a parameterization of SwiGLU. Patching the Python enum would have converted an AttributeError at config time into a kernel-not-found or a silently-wrong-math failure at launch time. A source build from any commit predating the artifact-hash bump would have compiled the host side happily and still had nothing to launch.

The timing is the other half of the story. The engine code that calls this kernel was written against a private build and carries an in-source comment referring to "the private cubin", and the engine's own pinned requirements point at a FlashInfer version that also predates SiTU. So nobody could have served this path from public wheels on day zero. It became possible three days later. The failure was a race, not a mistake.

Diagnostic method, in one line: read the traceback down to the exact symbol, verify that symbol against the public source at the exact version rather than assuming, then measure the layer nobody thinks to check.

Wall 2: a keyword argument that has never existed in public

Clearing wall 1 moved the failure one layer deeper, out of config and into the worker:

TypeError: trtllm_fp4_block_scale_moe() got an unexpected keyword argument (name withheld)

The obvious hypothesis is that a public API removed a parameter and there is a migration to follow. There is not. We AST-diffed every keyword argument at both call sites against the public function signature: 31 kwargs passed, exactly one absent, all 30 others matching by name. Then six searches across the public FlashInfer repository's code, pull requests and issues, and across TensorRT-LLM, returned total_count: 0 for the token in every case. It was never removed from public FlashInfer, because it was never in it. It is a parameter of a non-public build the day-0 branch was developed against, with no public ancestor, successor or synonym. Upstream vLLM's own public path for this model is the pinned file minus exactly those argument lines, with a hasattr capability probe in place of the assumption.

The reason this is worth writing down is not the fix. It is the risk analysis a skeptical reader should demand. Dropping an argument is only safe if it cannot silently select the wrong branch, and four independent reasons say it cannot. Public FlashInfer has exactly one implementation of that function, reached by a fixed positional call with no privacy, bundle or path selector. Cubin family selection is demonstrably activation_type, which is already passed, and which is provably the selector because bumping the release is what cleared wall 1. Artifact-bundle selection is a module-level string constant resolved once at import, the wrong shape for a per-call boolean to switch. And every remaining degree of freedom travels as an explicit tensor in the public signature. The failure mode of being wrong is a loud TypeError at the first forward pass, not silence.

One scare was checked and cleared on the way, because it looks exactly like a silent-wrong-math bug. FlashInfer's reference treats the first half of the FC1 output as the linear term and the second as the gate; vLLM's reference does the opposite. They are not in conflict: the same FlashInfer reference file applies the identical convention to its non-SiTU gated activations, so the ordering is universal to that kernel family and is handled once in weight preparation. What remains genuinely open is that the public reference is documented as "SiTU v2", v1 was private and cannot be read, and that risk exists with or without the keyword argument.

Wall 3: a CUDA graph capture crash, a mechanism, and a retraction

The shipped configuration disables the engine's custom all-reduce. That is not a taste preference. Remove the flag as the only variable, and eight workers print, at the end of FULL CUDA graph capture:

Failed: Cuda error /opt/vllm/csrc/custom_all_reduce.cuh:164 'invalid argument'

Lines 686 to 691 of results/raw/allreduce-armB-vllm.log; the arm carrying the flag was green end to end. The flag stays, and the per-layer communication tax is recoverable only by an upstream fix. The audit that priced that tax then corrected the price downward: the shipped run already dispatches symmetric-memory all-reduce ahead of NCCL (results/raw/k4pub-armB-vllm.log line 102), so the residual is small rather than the largest lever in the program.

A second, separate capture crash produced this project's cleanest example of a hypothesis being formed, appearing proven, and then being withdrawn. The W4A8 expert backend died at graph capture with cudaErrorIllegalState after a 36-minute FlashInfer autotune pass (results/raw/wa8r5-armB-vllm.log, error at line 742). The hypothesis was that failed autotune tactics poison the CUDA context. A follow-up window disabled autotune on both arms, and the backend that had died at capture served the entire suite under FULL graphs at 140.30 tok/s p50 single stream (results/raw/wa8r8-comparison.md). The campaign log recorded the mechanism as proven.

It then bought the confirming experiment, and the confirming experiment killed it. FlashInfer's own tactics-blocklist generator, run on the actual B300s at this model's real expert shape (896 experts, hidden 7168), found ZERO invalid tactics and produced no file. The serve that followed survived capture with autotune ON and no blocklist at all, at 140.42 tok/s p50 against 140.30 with autotune off (results/raw/blocklist-manifest.json, results/raw/blocklist-single-stream.json). One failure and two successes across two differing conditions attribute nothing. The mechanism claim is withdrawn in results.md. The honest statement is that this backend can serve under full CUDA graphs at about 140.4 tok/s, that the original crash is unexplained and possibly intermittent, and that turning autotune off costs nothing measurable, so it stays a free choice rather than a fix.

Walls 4 and 5, briefly

The FlashInfer MLA path requires the KV block count to satisfy an alignment condition against 128; engine init raises a ValueError otherwise, and the resolution is to serve with a block size of 128 or an aligned block-count override (docs/RUNBOOK-DROP.md, recorded from rehearsal on the proxy model and consistent with the resolved engine state in the committed serve logs).

The fifth wall frames all the others: for the measurement window described here, no released or nightly wheel served this model. That is not a claim about anyone's engineering. It is the ordinary gap between a merge and a release, and it is exactly the gap in which everything above had to be diagnosed.

What is still open

The W4A8 expert backend measures a 13 percent shorter step and is unshippable. Its distribution gate FAILED at top-1 65.94 percent, mean KL 0.5364, with 24.16 percent of positions skipped for thin shared support (results/raw/w4a8gate-gate-result.json), and the gate instrument itself is known unsound on this model at tensor parallel 8. The instrument that would settle it is task-level evaluation, which that tool reports as SKIPPED and unwired.

The unexplained capture crash has not been reproduced under repeated trials at one condition, because that has not been bought. Custom all-reduce stays disabled. No accuracy gate ran in the bring-up window at all. And the highest-concurrency point on record, 128, has no committed raw file behind it and therefore cannot be cited.