Optimization technique
Channelwise FP8 stores weights in eight-bit floating point and scales activations for each token at runtime.
We measured 2 published packages across 1 GPU target, using the recorded baseline and optimized conditions.
What made these builds shippable is not the FP8 itself but the measured accuracy verdict beside it, taken on the same protocol as the baseline.
We show absolute baseline and optimized values before the shared verdict. We keep each condition attached to its result.
| Model | Target | Throughput | Median latency | Verdict and conditions | Accuracy | Price and date |
|---|---|---|---|---|---|---|
| AREX-TurboBAAI/AREX-Turbo | H100vLLM 0.25.1 | 1534 tokens/s baseline1847 tokens/s optimized | 666 ms p50 baseline551 ms p50 optimized | 1.2x faster.baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. | Baseline 0.3821, optimized 0.4094.gsm8k no measurable accuracy change, passedStandard errors: baseline +/- 0.0134; optimized +/- 0.0135. | $15Verified Jul 27, 2026 |
| Qwythos-9B-Claude-Mythos-5-1Mempero-ai/Qwythos-9B-Claude-Mythos-5-1M | H100vLLM 0.25.1 | 974 tokens/s baseline1255 tokens/s optimized | 1058 ms p50 baseline815 ms p50 optimized | 1.29x faster.baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. | Baseline 0.8378, optimized 0.8324.gsm8k 99.35% recovery, passedStandard errors not published. | $20Verified Jul 25, 2026 |
We restate the published throughput pairs. The neutral fills do not declare a winner.
1.2x faster.
1.29x faster.
We do not call a score change an improvement when the gap is smaller than about twice the combined standard errors.
AREX-Turbo measured 0.3821 at baseline and 0.4094 optimized on gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2). No measurable change in accuracy. The difference between the baseline and optimized scores is smaller than the combined measurement error, so it is within measurement noise. Standard errors: baseline +/- 0.0134; optimized +/- 0.0135.
Qwythos-9B-Claude-Mythos-5-1M measured 0.8378 at baseline and 0.8324 optimized on gsm8k exact_match strict (N=1319, completion protocol). Measured difference, significance not published. The optimized score measured below the baseline, at 99.35% of it. Neither score published a standard error, so this difference cannot be separated from sampling noise. The difference is not stated as a regression. Standard errors not published.
Tasks beyond gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2) and gsm8k exact_match strict (N=1319, completion protocol) not published.
GPUs beyond H100 not published.
Serving engines beyond vLLM 0.25.1 not published.
We tie each accuracy verdict to its published task metric. AREX-Turbo: Throughput measured 1534 tokens/s at baseline and 1847 tokens/s optimized. P50 latency measured 666 ms at baseline and 551 ms optimized. 1.2x faster. AREX-Turbo measured 0.3821 at baseline and 0.4094 optimized on gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2). No measurable change in accuracy. Standard errors: baseline +/- 0.0134; optimized +/- 0.0135. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 27, 2026. Qwythos-9B-Claude-Mythos-5-1M: Throughput measured 974 tokens/s at baseline and 1255 tokens/s optimized. P50 latency measured 1058 ms at baseline and 815 ms optimized. 1.29x faster. Qwythos-9B-Claude-Mythos-5-1M measured 0.8378 at baseline and 0.8324 optimized on gsm8k exact_match strict (N=1319, completion protocol). Measured difference, significance not published. Standard errors not published. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 25, 2026.
We compare throughput and p50 latency under the recorded conditions. AREX-Turbo: Throughput measured 1534 tokens/s at baseline and 1847 tokens/s optimized. P50 latency measured 666 ms at baseline and 551 ms optimized. 1.2x faster. AREX-Turbo measured 0.3821 at baseline and 0.4094 optimized on gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2). No measurable change in accuracy. Standard errors: baseline +/- 0.0134; optimized +/- 0.0135. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 27, 2026. Qwythos-9B-Claude-Mythos-5-1M: Throughput measured 974 tokens/s at baseline and 1255 tokens/s optimized. P50 latency measured 1058 ms at baseline and 815 ms optimized. 1.29x faster. Qwythos-9B-Claude-Mythos-5-1M measured 0.8378 at baseline and 0.8324 optimized on gsm8k exact_match strict (N=1319, completion protocol). Measured difference, significance not published. Standard errors not published. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 25, 2026.
AREX-Turbo and Qwythos-9B-Claude-Mythos-5-1M are the published models on this page. AREX-Turbo: Throughput measured 1534 tokens/s at baseline and 1847 tokens/s optimized. P50 latency measured 666 ms at baseline and 551 ms optimized. 1.2x faster. AREX-Turbo measured 0.3821 at baseline and 0.4094 optimized on gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2). No measurable change in accuracy. Standard errors: baseline +/- 0.0134; optimized +/- 0.0135. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 27, 2026. Qwythos-9B-Claude-Mythos-5-1M: Throughput measured 974 tokens/s at baseline and 1255 tokens/s optimized. P50 latency measured 1058 ms at baseline and 815 ms optimized. 1.29x faster. Qwythos-9B-Claude-Mythos-5-1M measured 0.8378 at baseline and 0.8324 optimized on gsm8k exact_match strict (N=1319, completion protocol). Measured difference, significance not published. Standard errors not published. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 25, 2026.
AREX-Turbo: Throughput measured 1534 tokens/s at baseline and 1847 tokens/s optimized. P50 latency measured 666 ms at baseline and 551 ms optimized. 1.2x faster. AREX-Turbo measured 0.3821 at baseline and 0.4094 optimized on gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2). No measurable change in accuracy. Standard errors: baseline +/- 0.0134; optimized +/- 0.0135. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 27, 2026. Qwythos-9B-Claude-Mythos-5-1M: Throughput measured 974 tokens/s at baseline and 1255 tokens/s optimized. P50 latency measured 1058 ms at baseline and 815 ms optimized. 1.29x faster. Qwythos-9B-Claude-Mythos-5-1M measured 0.8378 at baseline and 0.8324 optimized on gsm8k exact_match strict (N=1319, completion protocol). Measured difference, significance not published. Standard errors not published. baseline used request profile chat-128, concurrency 8, and measured basis; optimized used request profile chat-128, concurrency 8, and measured basis. Verified Jul 25, 2026. Tasks beyond gsm8k exact_match (N=1319 full set, chat-template, deliberation-aware extraction v2) and gsm8k exact_match strict (N=1319, completion protocol) not published. GPUs beyond H100 not published. Serving engines beyond vLLM 0.25.1 not published.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
