Quantized and serving-tuned for H100 on vLLM 0.25.1, with the benchmark receipt included. You get a self-contained kit: the optimized weights, the exact serving configuration, and the measured proof. Run it on any cloud or your own hardware. Nothing calls home, nothing expires.
We keep re-optimizing and re-benchmarking as engines move, and you always download the latest verified version. Cancel anytime. Nothing is charged until you confirm at checkout.
Measured against the unoptimized baseline on H100: median latency 666 ms down to 551 ms. Concurrency 8. The full benchmark methodology and receipt ship inside the kit.
Runs on any GPU provider
The kit is self-contained: bring it to any cloud with H100-class GPUs, or your own hardware.
Docker Compose
Kubernetes
RunPod
Modal
Your own hardware
This is what it means to own your AI
You control the intelligence
The weights run on your infrastructure. No deprecation notices, no silent model swaps, no provider deciding what your product is allowed to do.
You keep the economics
One purchase instead of a meter that runs forever. A closed API charges you for every token, at whatever the price becomes; your own deployment costs what your GPU costs.
You keep the data
Prompts and outputs never leave your network. Nothing calls home, nothing is retained by someone else, and compliance stops being a negotiation.
llm-compressor 0.12.0, data-free: no calibration corpus is read at any point, so the quantization cannot have seen evaluation data
Quantization-sensitive layers held at source precision
the gated linear-attention (GDN) path and the vision tower are excluded from quantization and remain at BF16, which is the recipe measured lossless on this architecture family before it was reused here
Serving configuration swept per model and pinned
the exact engine build and arguments every receipt number was produced on: vLLM 0.25.1, max_num_seqs 128, max_num_batched_tokens 16384, max-model-len 4096, chosen because it beat the wider 256/32768 anchor on both latency and throughput for this model rather than inherited from another package
Published GPU capacity is not a proven minimum fit requirement.
Measured accuracy
gsm8k exact_match
(N=1319 full set, chat-template, deliberation-aware extraction v2)
Passed accuracy gate
Baseline
0.3821±0.0134 standard error
Optimized
0.4094±0.0135 standard error
Recovery
107.14%
Delta
+7.14%
No measurable change in accuracy
The difference between the baseline and optimized scores is smaller than the combined measurement error, so it is within measurement noise.
Methodology
NO MEASURABLE CHANGE, and specifically NOT an improvement. Measured on the full 1319-item gsm8k set on both sides: 0.3821 +/- 0.0134 baseline against 0.4094 +/- 0.0135 optimized. The gap is 1.43 sigma, inside the noise band, so the two builds are statistically indistinguishable on this instrument. The ratio reads as 107 percent recovery and that is noise: a quantized build does not gain reasoning ability, and any recovery above 100 percent is reported as indistinguishable rather than as a gain. INSTRUMENT NOTE, because it changes how this number should be read: lm-eval's own filters are not valid for a model whose chat template injects the thinking opener into the prompt. Its strict-match filter returns a structural 0.0, and its flexible-extract filter scores the last number in the response, which grades answer formatting rather than correctness. Measured example from this very comparison: on a problem whose answer is 18, both builds computed 18, but the BF16 response trailed off with '16' and scored wrong while FP8 ended on '18' and scored right. Under that filter the same two runs show a +0.0902 gap;
swapping only the extractor removed 70 percent of it. The figures above use a deliberation-aware extractor on both sides, with an unparsed rate of 0.000 baseline and 0.002 optimized, so extraction succeeded on essentially every sample. Protocol: seed 1234, zero-shot, chat template applied, 14000 generation budget, identical settings and the same H100 class on both sides, optimized side bound to the shipped artifact sha256 37cfa0aa. The gate verdict covers this gsm8k metric;
Buyer disclosure
What we did not measure
Refusal, toxicity and jailbreak behavior: the refusal-delta suite measured the CHANGE quantization caused against this model's own BF16 baseline on fingerprint-identical prompt sets and found it within tolerance. Absolute safety was not certified, and toxicity and jailbreak resistance were not evaluated. Run your own safety evaluation before user-facing or regulated deployment.
Peak VRAM is essentially unchanged: 71.20 GiB baseline against 71.35 GiB optimized, both measured at the same memory utilization target. This is a bandwidth win, not a memory-footprint win, because the weight saving is absorbed by KV cache at fixed utilization. Do not expect to fit a larger batch purely from the FP8 weights.
Text-only. The source checkpoint ships a real vision tower and its weights are carried through unmodified at BF16, but image behavior was not evaluated and is not certified by the benchmark receipt.
The gsm8k figure depends on the answer extractor, and no available extractor is perfect for this model because it frequently emits no closing thinking tag. Both sides were scored identically with a deliberation-aware extractor and the unparsed rate was under 0.2 percent, but treat the absolute score as instrument-bound rather than as this model's definitive gsm8k number.
License
Two licenses apply: the RunInfra package license you purchase under, and the base model's own open-source license.
Package license
RunInfra Package License v1.1 (2026-07-25)
Buy once. No meter, no expiry, nothing calling home. The full terms ship inside the kit.
What you are licensed to do
This package is licensed to the purchasing workspace, one time, for the exact version purchased. Everyone in that workspace may run it in production without limits: unlimited inference, on any hardware or cloud the workspace controls, for any lawful commercial purpose. You may modify the configuration and tooling for your own use.
Your outputs are yours
Everything the model produces for you belongs to you. You may use, sell, and build products on the model's outputs without restriction or royalty. Serving the model to your own customers as part of your product is use, not redistribution, and is fully allowed.
What you may not do
You may not redistribute, resell, sublicense, rent, publish, or otherwise make the package or its artifacts available to any third party. That covers the optimized weights, serving configuration, scripts, kit archive, and benchmark receipts, whole or in part, modified or not. The license belongs to the purchasing workspace and cannot be transferred separately from it.
The base model keeps its own license
The underlying model remains under its upstream open-source license, which is included in this kit with attribution and a statement of RunInfra's modifications. Nothing in this license restricts rights the upstream license grants you for the ORIGINAL model; the restrictions above apply to RunInfra's optimized package.
Version-pinned, as measured
You purchased this exact version, proven against the exact engine version named in the receipt. It stays downloadable to your workspace and never expires. The benchmark receipt describes measurements taken at verification time on the named hardware; RunInfra does not promise future updates to this version, and later package versions are separate purchases.
Breach ends the license
If the workspace redistributes the package or its artifacts, this license terminates for that workspace. Sections about your outputs survive termination for outputs already produced.
Base model license
Apache-2.0, as published by the model author on Hugging Face. The full license text ships inside the kit.
ifeval, tool calling and safety refusal delta are recorded separately in accuracy-merged.json and all three returned a gate-eligible state with no exemption.
IFEval: instruction-level pass was 112 of 150 baseline against 119 of 153 optimized. The scoreable denominators differ slightly because terminal exclusions differ per side, so this is a paired no-measurable-change signal rather than a publishable absolute score comparison.
Tool calling: 8 of 8 on the RunInfra battery with every metric at 1.0 on BOTH sides, using the qwen3_coder parser, which is the grammar this model family actually emits.
Minimum VRAM is listed as 80 GB because that is what the shipped kit manifest records and the kit is what your verifier checks against. It describes the card this package was measured on, not the model's requirement: the FP8 weights are 6.27 GB. The package has only been characterized on H100, so smaller cards are untested rather than unsupported.
Measured at 4096 context, concurrency 8, on a single H100 with a per-model swept serving configuration (max_num_seqs 128, max_num_batched_tokens 16384, chosen because it beat the 256/32768 anchor on both latency and throughput). Longer context, other GPUs and higher concurrency are not characterized, though a soak at concurrency 48 completed 256 of 256 requests with zero failures on both sides at 5804.5 output tokens per second optimized against 5076.0 baseline.
Speculative decoding is NOT enabled and cannot be: the model config declares a multi-token-prediction layer but the checkpoint ships zero draft-head weights, so enabling it would build a randomly initialized head with near-zero acceptance. This was caught by reading the safetensors index at preflight.
Calibration
No contamination risk: fp8_channel_dynamic is DATA-FREE (round-to-nearest weights plus dynamic per-token activation scales). No calibration corpus is read at any point, so overlap between calibration data and evaluation data is structurally impossible rather than merely unlikely.
Cost basis
Cost inputs current as of 2026-07-27; economics at the measured operating point: 1847.4 output tokens/sec on one H100 at the chat-128 profile, concurrency 8, which at an assumed USD 4.50/hr is about USD 0.68 per 1M output tokens. At soak-48 saturation (5804.5 tokens/sec) it is about USD 0.22 per 1M. Throughput is measured; the hourly rate is an assumption, so recompute for your own rate.
RunInfra is a sub-product of RightNow Research Lab.
Y CombinatorBacked by Y Combinator.
RunInfraby RightNow
