# RunInfra (full text)

> RunInfra serves open models for agents and apps through OpenAI-compatible and Anthropic-compatible Model APIs at https://api.runinfra.ai/v1, billed per token. Setup guides connect coding agents such as Claude Code and Codex. Backed by Y Combinator.

## Model APIs

Model APIs is RunInfra's hosted model API product at https://api.runinfra.ai/v1. Each model publishes its operation and compatibility. Chat models expose OpenAI-compatible chat completions at POST /v1/chat/completions and Anthropic-compatible Messages at POST /v1/messages. Chat model requests use workspace API keys and support streaming. JSON mode is supported per chat model, and a model that requires a specific reasoning_effort to deliver it publishes that value with its capabilities. Pricing is published per model in USD with only the billed dimensions shown. This file also contains technical reference material, GPU reference data, and newsroom articles selected for answer-engine ingestion without rendering the UI.

The entries below come from the hosted model registry. A registered model appears here only when its own public row passes the same publication gate used by the sitemap. Entries that do not pass that public gate are omitted.

- Model API library (https://runinfra.ai/inference-api): available hosted models with their operation and compatibility contracts.
- Nemotron 3.5 Lightning 30B Model API (https://runinfra.ai/inference-api/nemotron-3-5-lightning-30b): Served model ID: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.05 per 1M input tokens; $0.01 per 1M cached input tokens; $0.15 per 1M output tokens. Context window: 262,144 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text only.
- Qwen3.8 27B Model API (https://runinfra.ai/inference-api/qwen3-8-27b): Served model ID: Qwen/Qwen3.8-27B. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.10 per 1M input tokens; $0.01 per 1M cached input tokens; $0.40 per 1M output tokens. Context window: 262,144 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text and images.
- Ornith 1.5 35B Model API (https://runinfra.ai/inference-api/ornith-1-5-35b): Served model ID: ornith-ai/Ornith-1.5-35B-A3B. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.10 per 1M input tokens; $0.01 per 1M cached input tokens; $0.40 per 1M output tokens. Context window: 262,144 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text and images.
- GLM 5.3 Model API (https://runinfra.ai/inference-api/glm-5-3): Served model ID: zai-org/GLM-5.3. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $1.30 per 1M input tokens; $0.13 per 1M cached input tokens; $4.20 per 1M output tokens. Context window: 1,048,576 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text only.
- GLM 5.3 Flash Model API (https://runinfra.ai/inference-api/glm-5-3-flash): Served model ID: zai-org/GLM-5.3-Flash. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Input and output are 51% off until Nov 4, 2026, 7:59 AM UTC. Standard rates resume automatically: $0.11 per 1M input tokens, $0.03 per 1M cached input tokens, $0.45 per 1M output tokens. Context window: 1,048,576 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text and images.
- DeepSeek V4.1 Flash Model API (https://runinfra.ai/inference-api/deepseek-v4-1-flash): Served model ID: deepseek-ai/DeepSeek-V4.1-Flash. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.14 per 1M input tokens; $0.03 per 1M cached input tokens; $0.58 per 1M output tokens. Context window: 1,048,576 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text only.
- DeepSeek V4.1 Flash 1017 Model API (https://runinfra.ai/inference-api/deepseek-v4-1-flash-1017): Served model ID: deepseek-ai/DeepSeek-V4.1-Flash-1017. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.15 per 1M input tokens; $0.01 per 1M cached input tokens; $0.60 per 1M output tokens. Context window: 1,048,576 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text and images.
- Ornith 1.5 122B Model API (https://runinfra.ai/inference-api/ornith-1-5-122b): Served model ID: ornith-ai/Ornith-1.5-122B. OpenAI-compatible chat completions: POST /v1/chat/completions. Anthropic-compatible Messages: POST /v1/messages. Published token pricing: $0.26 per 1M input tokens; $0.07 per 1M cached input tokens; $2.08 per 1M output tokens. Context window: 262,144 tokens. Published capabilities: Tool calling, JSON mode, Streaming. API access is available.
  Accepted input: Text and images.

## Coding agents

- [Set up with your coding agent](https://runinfra.ai/docs/tools-sdks/agent-setup): Your coding agent sets up RunInfra's open models after you approve the sign-in and the changes it proposes.

Coding agents: Your coding agent sets up RunInfra's open models by following the agent setup guide. It shows the proposed changes, applies them only after approval, and verifies them with a real API request that counts toward workspace usage. The per-agent guides are references for configuration, compatibility limits and recovery.

### Set up RunInfra with your coding agent

Set up RunInfra's open models in this coding agent for me. Read https://runinfra.ai/docs/tools-sdks/agent-setup.md and follow it step by step. If you cannot open links, run `npx -y @runinfra/cli@latest --help` and follow its FOR CODING AGENTS section. I approve sign-in in my browser. Before you install anything, change this agent's settings or spend money, show me the plan and wait for my yes.

- [Claude Code](https://runinfra.ai/docs/tools-sdks/claude-code): Anthropic-compatible Messages setup and agent-specific guidance.
- [Codex](https://runinfra.ai/docs/tools-sdks/codex): Separate Codex command setup and Responses provider reference.
- [RightNow Agent](https://runinfra.ai/docs/tools-sdks/rightnow): Native RunInfra provider setup.
- [OpenCode](https://runinfra.ai/docs/tools-sdks/opencode): Provider and model setup guide.
- [Pi](https://runinfra.ai/docs/tools-sdks/pi): Provider and model setup guide.
- [Aider](https://runinfra.ai/docs/tools-sdks/aider): OpenAI-compatible configuration guide.
- [Qwen Code](https://runinfra.ai/docs/tools-sdks/qwen-code): Provider and model selection guide.
- [Kilo Code](https://runinfra.ai/docs/tools-sdks/kilo-code): Compatible provider configuration guide.
- [Goose](https://runinfra.ai/docs/tools-sdks/goose): Provider setup and credential handoff guide.
- [Continue](https://runinfra.ai/docs/tools-sdks/continue): Configuration and restart guidance.
- [Cline](https://runinfra.ai/docs/tools-sdks/cline): Provider setup and manual IDE verification.
- [Droid](https://runinfra.ai/docs/tools-sdks/droid): Personal model configuration and selection guide.
- [Zed](https://runinfra.ai/docs/tools-sdks/zed): Preview setup and manual credential handoff limitations.
- [Grok Build](https://runinfra.ai/docs/tools-sdks/grok-build): Provider and model picker guide.
- [Hermes](https://runinfra.ai/docs/tools-sdks/hermes): Custom provider setup guide.
- [OpenClaw](https://runinfra.ai/docs/tools-sdks/openclaw): Provider setup and model selection guide.
- [Crush](https://runinfra.ai/docs/tools-sdks/crush): Add RunInfra alongside your provider, with configuration and recovery references.

- [Docs MCP server](https://runinfra.ai/docs/mcp): Retrieve RunInfra documentation from an MCP client.
- [Docs MCP server card](https://runinfra.ai/docs/.well-known/mcp/server-card.json): Published documentation-server discovery metadata.

## Pricing

Pay as you go: Model API usage is billed per token at each model's published input, cached-input where configured, and output prices. Only the billed dimensions are shown. Check the model page for current availability and prices.

- [Pricing](https://runinfra.ai/pricing): current payment options.

- [Pricing docs](https://runinfra.ai/docs/introduction/plans): credits, top-ups, and usage billing.

### Coding plans

- [Coding plans](https://runinfra.ai/pricing#coding-plans): Starter: $10 a month. About $12 of usage a month, paced as $2.77 a week and $1.11 per 5 hours. Pro: $29 a month. About $35 of usage a month, paced as $8.08 a week and $3.23 per 5 hours. Team: $99 a month. About $119 of usage a month, paced as $27.46 a week and $10.98 per 5 hours. At a limit, keep full speed on credits (your choice: a cap, or no limit), then Standby, free and slower. Images and audio always stay pay as you go.

Shared by this workspace, not per member

Q: What happens at a limit?
A: If you turned on Keep full speed on credits, credits pay with your choice of a cap per billing period or no limit. When credits cannot pay, free Standby serves chat. Choose at checkout and change it in Coding plan. If neither can serve a request, it stops with the reason, reset time and next step.

Q: What does my plan cover?
A: Chat, through chat completions, messages and responses. For available Model APIs, from every workspace key except keys marked Credits only. Images and audio always stay pay as you go.

Q: Can I cancel?
A: Anytime in Billing. Your plan ends at the period end, or at once while a payment dispute is open. Deleting your account ends it at once, with no refund of the current month. Afterwards your keys keep working on credits, which keep their expiry dates.

Q: When do the limits reset?
A: Your 5-hour window starts with your first plan request, and the next request after it ends starts a new one. The weekly limit resets on a fixed schedule from activation. The plan pays while both have room.

Q: How is usage counted?
A: At our standard per-token prices, with cached input at the cached price. Plan usage is not a credit deposit and does not roll over. Monthly usage is approximate: months hold different numbers of weekly resets.

Q: What is Standby?
A: Free, slower service when we are busy, up to a daily amount. Standby covers chat only. It refills at midnight UTC, may be reduced or paused, and is off while a plan payment is due or processing, or credits are owed.

Q: When am I charged?
A: At the start of each billing period. Your plan renews monthly until canceled. Upgrades apply now with a prorated charge, and counted usage carries over. Downgrades apply at the period end.

Q: What can I count on?
A: Requests that fail before returning output never use your plan. Client cancellations count usage, as in pay as you go. Each response header shows whether your plan, credits or Standby paid. The same payment source covers the request from start to finish. Metering changes come with written notice, and promotional rates are labeled temporary.

Q: What if a payment fails or is disputed?
A: After a failed renewal, your plan can keep serving for up to 48 hours, or 7 days if it has had a paid renewal. A disputed payment pauses the plan and ends it at the period end.

## Documentation

- [Docs](https://runinfra.ai/docs): Product documentation and integration guides.
- [Model APIs quickstart](https://runinfra.ai/docs/api-reference/model-apis-quickstart): Make a first call with a workspace API key.
- [Authentication](https://runinfra.ai/docs/api-reference/authentication): Workspace API keys and request authentication.
- [Chat Completions API reference](https://runinfra.ai/docs/api-reference/chat-completions): OpenAI-compatible chat request and response contract.
- [Anthropic Messages API reference](https://runinfra.ai/docs/api-reference/anthropic-messages): Anthropic-compatible request and response contract.
- [Model discovery](https://runinfra.ai/docs/api-reference/models): Discover model IDs and their published capabilities.
- [OpenAI compatibility](https://runinfra.ai/docs/tools-sdks/openai-compatibility): Chat Completions, Responses, client setup, and compatibility limits.
- [OpenAPI specification](https://runinfra.ai/docs/api-reference/openapi.json): Machine-readable API reference. Current product availability is listed on each model page.
- [Errors](https://runinfra.ai/docs/api-reference/errors): Error codes, causes, and recovery guidance.
- [Rate limits](https://runinfra.ai/docs/api-reference/rate-limits): Request limits and retry guidance.
- [Prompt caching](https://runinfra.ai/docs/cookbook/prompt-caching): Prompt reuse for agents, cache usage, and cached-input pricing.
- [Session affinity](https://runinfra.ai/docs/api-reference/session-affinity): Session routing and its limits for repeated agent requests.
- [Idempotent retries](https://runinfra.ai/docs/api-reference/idempotent-retries): Retry behavior and request identity.
- [Docs llms.txt](https://runinfra.ai/docs/llms.txt): Documentation index for agents, with Markdown page links.
- [Docs sitemap](https://runinfra.ai/docs/sitemap.xml): Canonical documentation pages.

## Evidence and methodology

- [Methodology](https://runinfra.ai/methodology): definitions, measurement conditions, sources, and limitations for published metrics.
- [Benchmarks](https://runinfra.ai/benchmarks): historical self-reported serving results with incomplete runtime and workload provenance, source and date beneath every chart. The source record belongs to the reported workload; historical results do not establish current Model API performance.

Published model pages own current prices, availability, context limits, and capabilities. Cache measurements describe their stated observation window, not a guaranteed result for a new workload.

### Capability boundaries

Model API availability and supported operations are listed on each model page. Technical references and benchmark results do not imply that other models or workflows are available as products.

## Optional

Reference material below retains its own scope, evidence, and dates; it does not expand the current product offering.

### Technical reference

Optional model-selection, GPU, benchmark, and self-hosting reference material.

Key pages: Model APIs and technical overview (https://runinfra.ai), inference cost calculator (https://runinfra.ai/calc, server-rendered fit plus infrastructure economics and measured RunInfra evidence only when the required source rows exist), benchmarks (https://runinfra.ai/benchmarks, historical self-reported serving results with incomplete runtime and workload provenance, source and date beneath every chart), security posture (https://runinfra.ai/security, SOC 2 Type II), research papers (https://runinfra.ai/research), documentation (https://runinfra.ai/docs, AI index at https://runinfra.ai/docs/llms.txt).

- [Inference cost calculator](https://runinfra.ai/calc): workload-specific fit and economics, with source evidence where published.
- [Glossary index](https://runinfra.ai/glossary): definitions of serving mechanisms and metrics.
- [Comparisons](https://runinfra.ai/compare): historical comparisons with their workload conditions and limitations.
- [Full reference index](https://runinfra.ai/llms.txt): model, engine, GPU, research, and policy references.

### GPU reference

- NVIDIA L4, 24 GB VRAM
- NVIDIA L40S, 48 GB VRAM
- NVIDIA A100, 80 GB VRAM
- NVIDIA H100, 80 GB VRAM
- NVIDIA H200, 141 GB VRAM
- NVIDIA B200, 192 GB VRAM

### Newsroom articles

---

#### RunInfra is now inference for agents

URL: https://runinfra.ai/news/inference-for-agents
Category: Announcement
Published: September 26, 2026
Author: Jaber Jaber (Founder and researcher, RunInfra)
Read time: 2 min read
Tags: Model APIs, hosted inference, coding agents

RunInfra now focuses on hosted inference for agents, with pay-as-you-go Model APIs and setup guides for Claude Code, Codex and other coding agents.

##### Frequently asked

Q: Where should I start?
A: Choose a model in the model library, then follow the Model APIs quickstart.

Q: Can I connect Claude Code or Codex?
A: Yes. Follow the agent-specific setup guide, including any manual steps it requires.

Q: Do historical benchmarks establish current Model API performance?
A: No. Historical results do not establish current Model API performance.

##### References

- Model library: https://runinfra.ai/inference-api
- Model APIs quickstart: https://runinfra.ai/docs/api-reference/model-apis-quickstart
- Claude Code setup: https://runinfra.ai/docs/tools-sdks/claude-code
- Codex setup: https://runinfra.ai/docs/tools-sdks/codex
- Historical benchmarks: https://runinfra.ai/benchmarks

---

#### With software alone, one B200 beats the LPU and gets close to Cerebras

URL: https://runinfra.ai/news/b200-beats-the-lpu
Category: Engineering
Published: August 6, 2026
Updated: August 9, 2026
Author: Jaber Jaber (Founder and researcher, RunInfra)
Read time: 23 min read
Tags: B200, gpt-oss-120b, SGLang, Speculative decoding, Cerebras, Groq, Benchmark

Stock SGLang served gpt-oss-120b at 411 tokens per second on one rented B200. Four settings took the same card and the same weights to 1,366, and single prompts to 2,215. No kernel written, nothing recompiled.

##### Frequently asked

Q: How fast can one B200 serve gpt-oss-120b with software changes alone?
A: 411 tokens per second on stock SGLang 0.5.16, and 1,165 to 1,366 tokens per second after four configuration changes on the same card with the same weights. Individual prompts reached 2,215. No kernel was written and nothing was recompiled.

Q: Does a GPU beat Groq or Cerebras on gpt-oss-120b?
A: It beats Groq and SambaNova and does not beat Cerebras. One tuned B200 measured 1,165 to 1,366 tokens per second against Groq at 477 and SambaNova at 706, which is 2.44x to 2.87x faster than Groq. Cerebras publishes 1,891, so the tuned GPU reaches about 62 to 72 percent of it.

Q: What is the ceiling for gpt-oss-120b on a B200?
A: About 1,610 tokens per second for single-stream decode. Decoding one token reads every active weight once, which is 1.91 GB of attention in bf16, 1.90 GB of top-4 routed experts in mxfp4 and 1.16 GB of lm_head in bf16, so 4.97 GB per token. Against 8 TB/s of HBM3e that is a floor of 0.621 milliseconds per token.

Q: Which SGLang settings mattered most for single-stream decode?
A: Widening the n-gram search was worth the most by far. speculative_ngram_max_bfs_breadth ships at 10, and raising it to 24 took the same configuration from 697 to 1,165 tokens per second. Past 24 the CPU search stops paying for itself and throughput falls off a cliff, to 341 at 48.

Q: Does speculative decoding change the output?
A: Not under greedy verification. A drafted token is accepted only where it matches the token the target model would have produced, and on the first mismatch the remaining draft is discarded and the target's token is used. Speculation changes how many forward passes you spend, not which tokens come out.

Q: Why do the same settings give different numbers per run?
A: N-gram speculation drafts by matching text it has already produced, so acceptance depends on how predictable the output is and on what the trie has built up. The same configuration measured 1,165.1 and 1,366.2 across two runs, and per prompt ranged from 219 on open-ended prose to 2,215 on structured reasoning.

##### References

- Cerebras, Blackwell vs Cerebras: https://www.cerebras.ai/blog/blackwell-vs-cerebras
- Artificial Analysis, gpt-oss-120b providers, read 9 August 2026; original chart images captured 6 August 2026: https://artificialanalysis.ai/models/gpt-oss-120b/providers
- Artificial Analysis methodology, Output Speed: https://artificialanalysis.ai/methodology
- SGLang speculative decoding documentation: https://docs.sglang.io/docs/advanced_features/speculative_decoding
- NVIDIA DGX B200 specifications: https://www.nvidia.com/en-us/data-center/dgx-b200/
- openai/gpt-oss-120b on Hugging Face: https://huggingface.co/openai/gpt-oss-120b

---

#### Lossless Inference

URL: https://runinfra.ai/news/lossless-inference
Category: Research note
Published: August 3, 2026
Author: Jaber Jaber (Founder and researcher, RunInfra)
Read time: 12 min read
Tags: LLM inference, Quantization, Speculative decoding, FlashAttention, KV cache, Serving

How to make LLM serving faster without touching the model. Exact kernels, speculative decoding, lossless compression, KV reuse and scheduling, with the math and how to verify it with logit parity.

##### Frequently asked

Q: What does lossless inference mean?
A: Bit exact means the optimized stack returns the same logits as the reference on every input, f_opt(x) = f_ref(x). Kernels and fusion, compilation, KV reuse, scheduling and lossless weight compression all live in this tier because they change how the math runs without changing the math itself

Q: Why does quantization hurt long agentic runs?
A: If you hold p at half a percent instead, survival over 20 steps stays at 0.90, and that is the difference between an agent you trust and an agent you babysit. The model never got visibly dumber, it just got 2 percent noisier per decision and the compounding did the rest

Q: How do you verify a lossless claim?
A: Lossless is a claim you can test, and I test it with logit parity. You run the same prompts through the reference stack and through yours with greedy decoding, then compare the maximum absolute logit difference per token, where zero is the pass bar for a bit exact claim and a distribution test is the bar for speculative decoding. I published this for bonsai-turbo, where 32 of 32 prompts pass against the vendor fork across all four engine configurations (github.com/RightNow-AI/bonsai-turbo). When parity fails the diff shows me where to look, and it is usually a fused kernel that cut a corner, a RoPE mismatch or a sampler that renormalizes differently

---

#### vLLM vs SGLang vs TensorRT-LLM: a serving benchmark

URL: https://runinfra.ai/news/vllm-vs-sglang-vs-tensorrt-llm-benchmark
Category: Engineering
Published: June 20, 2026
Updated: September 21, 2026
Author: Jaber Jaber (Founder and researcher, RunInfra)
Read time: 11 min read
Tags: vLLM, SGLang, TensorRT-LLM, Benchmark, LLM inference, GPU

Self-reported Llama-3.1-8B results for three LLM inference engines. Published unique-prompt throughput and latency rows cover H100 bf16. FP8 and L40S appear only in derived cost results. The harness and raw data are not publicly available.

##### Frequently asked

Q: vLLM or SGLang?
A: On plain unique-prompt traffic they are within a few percent, with vLLM slightly ahead on throughput, latency, and cost in our tests. Use vLLM as the default; reach for SGLang when its programmable frontend fits your workload.

Q: Is TensorRT-LLM worth it?
A: For latency-sensitive serving below its throughput ceiling, yes. It held the lowest time to first token across the mid range, 235 ms at concurrency 32 versus vLLM's 514 ms. We ran the PyTorch backend, the only execution backend in TensorRT-LLM 1.2.1.

Q: What happened to TGI?
A: Hugging Face moved Text Generation Inference to maintenance mode in December 2025 and archived the repository on March 21, 2026. We note it for context and did not benchmark it as a contender.

Q: Which GPU should I use?
A: For reported derived GPU cost on Llama-3.1-8B, H100 is lower in the published bf16 rows. With vLLM, the figures are $0.206 per 1M output tokens on H100 and $0.425 on L40S. No L40S throughput or latency rows are published, so these results do not support a speed comparison.

Q: Does the ranking hold at fp8?
A: The published fp8 results compare derived GPU cost only: $0.158 per 1M output tokens for vLLM and $0.17 for SGLang on H100. The published fp8 results provide no throughput or latency rows. They do not establish a speed ranking.

##### References

- TensorRT-LLM 1.2 release notes, TensorRT backend removal: https://nvidia.github.io/TensorRT-LLM/release-notes.html#tensorrt-llm-release-1-2
- Hugging Face TGI repository, maintenance since December 2025, archived March 21, 2026: https://github.com/huggingface/text-generation-inference
- PagedAttention (vLLM), Kwon et al., SOSP 2023, arXiv:2309.06180: https://arxiv.org/abs/2309.06180
- RadixAttention (SGLang), Zheng et al., NeurIPS 2024, arXiv:2312.07104: https://arxiv.org/abs/2312.07104
- Orca: continuous batching, Yu et al., OSDI 2022: https://www.usenix.org/conference/osdi22/presentation/yu

---