Chat completions

POST https://api.runinfra.ai/v1/chat/completions

This is the endpoint every Model APIs caller reaches. Send an OpenAI-compatible chat completion with one of the live model ids.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Explain exponential backoff."}],
    max_tokens=16384,
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const response = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [{ role: "user", content: "Explain exponential backoff." }],
  max_tokens: 16384,
});
console.log(response.choices[0]?.message?.content);

curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Explain exponential backoff."}],"max_tokens":16384}'

What each model supports

Requesting a capability a model does not have returns 400 with error.code hosted_capability_not_supported and param naming the field. You find out at the call, not in the output.

Give reasoning models room

Every live model except DeepSeek V4 Pro reasons before it answers by default; DeepSeek V4 Pro returns its answer directly. Reasoning tokens are billed output and they count toward max_tokens, ahead of the visible answer. A small budget can be spent entirely on thinking and return a completion whose content is empty, billed for the tokens it generated.

Measured on this endpoint on 2026-08-14 by sending the prompt “Write a TypeScript retry helper.” and recording how often a complete answer came back:

Model	Use at least	What smaller budgets did
`deepseek-v4-flash`	16,384	At 2,048 and 4,096 the answer was empty every time. At 8,192 it was empty half the time.
`nemotron-3-5-lightning-30b`	4,096	At 2,048 the answer was empty in 2 of 5 runs and cut off mid-sentence in 4 of 5.

The budget is a ceiling, not a spend. At 16,384 DeepSeek V4 Flash still stopped on its own after roughly 7,500 output tokens, and raising the ceiling further did not make it generate more. Setting the ceiling too low is what costs money, because the request is billed for the reasoning and returns nothing to show for it.

How to detect an exhausted budget in code

RunInfra preserves finish_reason: "length" and adds choices[].runinfra.output_status when a choice reaches its generation limit before producing final answer content.

{
  "choices": [
    {
      "finish_reason": "length",
      "message": { "content": null },
      "runinfra": {
        "output_status": {
          "code": "generation_limit_reached_before_answer",
          "message": "The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."
        }
      }
    }
  ],
  "usage": {
    "completion_tokens": 300,
    "runinfra": {
      "output_token_accounting": {
        "visible_answer_tokens": 0,
        "non_answer_completion_tokens": 300,
        "sources": {
          "visible_answer_tokens": "proxy_classified",
          "non_answer_completion_tokens": "provider_reported"
        }
      }
    }
  }
}

Branch on choices[].runinfra.output_status.code. generation_limit_reached_before_answer means the choice ended for length. If another terminal finish_reason produced no answer content, the code is no_answer_content and you should inspect finish_reason before retrying.The aggregate usage annotation appears only when every choice lacks a final answer. RunInfra does not estimate a reasoning-token count: completion_tokens stays the provider-reported total that settlement uses, and non_answer_completion_tokens records that those tokens produced no answer. The annotation changes no pricing.

Turning reasoning off

reasoning_effort: "none" turns reasoning off for one request. The completion then carries the answer and no reasoning stream. Measured on this endpoint on 2026-08-16 with an identical prompt at temperature 0, the completion fell from 35 tokens to 3 on deepseek-v4-flash, from 335 to 5 on nemotron-3-5-lightning-30b, and from 67 to 5 on qwen3-8-27b. Because reasoning bills at the output rate, this is the largest per-request cost lever on short tasks. qwen3-8-2-4t-a95b is the one model that cannot turn reasoning off. It refuses with 400 and the message “Disabling thinking is not supported.”, never silently ignored.

How the effort levels behave between none and the ceiling

Measured on 2026-08-16 at temperature 0 with repeated runs per level.

deepseek-v4-flash applies max when you send no reasoning_effort, and sends high and xhigh upstream as max. medium is reproducibly distinct from the default.
qwen3-8-2-4t-a95b produces reproducibly distinct reasoning at low, medium and xhigh (113, 140 and 89 completion tokens on the probe, with xhigh equal to the omitted default).
qwen3-8-27b measurably alters its reasoning at medium, against a twice-identical baseline.
nemotron-3-5-lightning-30b level deltas stayed inside the model’s own run-to-run variance, so treat effort there as the off switch only.

The vendor spellings enable_thinking, chat_template_kwargs.enable_thinking, thinking_budget and thinking are accepted for request-shape compatibility and have no effect on any listed model: a request relying on them still reasons and still bills for it. reasoning_effort is the one honored spelling.

What the gateway does with a field you send

Every field lands in one of three dispositions. These are checked against the contract below, then forwarded:

Field	Type	Default	Behavior
`model`	string	none	Required. Must resolve to a model your key can reach.
`messages`	object array	none	Required, at least one message. Extra message fields pass through.
`stream`	boolean	`false`	`true` returns an SSE response.
`temperature`	number, 0 to 2	none	Range checked, then forwarded.
`top_p`	number, 0 to 1	none	Range checked, then forwarded.
`max_tokens`	integer, 1 or more	see below	Per-generation output limit, counts toward the total output budget.
`max_completion_tokens`	integer, 1 or more	see below	Same. If you send both, the larger is used for the budget check.
`n`	integer, 1 to 4	`1`	Generated choices. The total output budget covers all of them.
`best_of`	integer, 1 to 4	`1`	Forwarded when present, included in the choice budget.
`stop`	string or string array	none	At most 4 sequences, 256 characters each. Beyond either bound returns `400`.

A message needs a non-empty string role. content may be a string, an array, or null. An assistant message needs content or a non-empty tool_calls. A tool message needs content and a non-empty string tool_call_id. Every other role needs content.

Sent untouched: the forwarding allowlist

The gateway accepts any JSON value on these, applies no default, and forwards it unchanged. The model can still reject the value, which comes back as 400 invalid_request_error.audio, frequency_penalty, function_call, functions, logit_bias, logprobs, metadata, modalities, moderation, parallel_tool_calls, prediction, presence_penalty, safety_identifier, seed, service_tier, store, tool_choice, top_logprobs, user, verbosity, web_search_options.Four of them do carry gateway behavior:

Field	Behavior
`reasoning_effort`	Forwarded unchanged, except where a model publishes a default or a rewrite. On `deepseek-v4-flash`, omitting it applies `max`, and `high`, `xhigh` and `max` all go upstream as `max`.
`response_format`	Forwarded to a model that can enforce it. On a model that cannot enforce a format while reasoning, `json_schema` and `json_object` are refused unless the request also sends `reasoning_effort: "none"`.
`stream_options`	The gateway always requests usage upstream so it can settle. Client-visible usage needs `include_usage: true`.
`tools`	Forwarded unchanged. Forwarding does not guarantee the model accepts a particular tool definition.

Removed before the request leaves the gateway

The allowlist is exhaustive: every other top-level field is dropped. Common examples are max_output_tokens, reasoning, text, include, truncation and previous_response_id (Responses API fields on the Chat route); top_k, min_p, repetition_penalty, length_penalty and use_beam_search; guided_json, guided_regex, guided_choice and guided_grammar; and num_return_sequences, num_beams and beam_width, which the gateway reads for the four-choice safety check and then removes.An unknown field can pass request validation and still be removed here, so never rely on silent pass-through for a field that is not on the allowlist.

Output limits

Limit	Value
Total output tokens per request	The model’s remaining context, that is its context window minus your prompt, across all generated choices. Oversized values are clamped, never refused.
Generated choices per request	4
Full response	240 seconds, then `504` `hosted_inference_response_timeout`
First token on a stream	60 seconds
JSON request body	3.5MB, then `413` `payload_too_large`
`stop` sequences	4, of 256 characters each

Omit both output-token fields and the gateway forwards no output limit at all, so the model generates until it stops on its own or reaches the remaining context. Send an oversized max_tokens and the gateway clamps it to what the context allows rather than refusing the request: the response carries X-RunInfra-Output-Clamped: true and X-RunInfra-Output-Token-Ceiling with the value used. Two bounds apply at once and the smaller one wins. Tokens are bounded by the remaining context above; wall-clock is bounded by the 240 second response limit, which at typical decode rates is the binding limit for very long generations. /v1/models publishes both, max_output_tokens beside response_time_ceiling_seconds, so size a long generation against the pair rather than the token figure alone. The 504 message names the budget that was exceeded and carries Retry-After: 1. For a non-streaming request, error.runinfra.remedies suggests stream: true, because response headers arrive with the first token, and a smaller max_tokens value. An already-streaming first-token timeout does not repeat the streaming remedy. A model’s context window is a separate per-model limit, published on that model’s page in the Model Library. Served context is a property of the deployment rather than of the weights, so read it there rather than assuming a figure from a model card.

Three things a request can be refused for

Each one is refused up front with a 400 naming the field, rather than answered with output you cannot use.

A JSON format the model cannot hold while reasoning

response_format of json_schema or json_object is enforced during generation, and on some models that enforcement cannot start until the model has finished reasoning. Those models return 400 with code hosted_parameter_not_supported and param: "response_format", so you pay for a parameter edit instead of for output your parser rejects.deepseek-v4-flash and nemotron-3-5-lightning-30b are affected today. On each, the fix is reasoning_effort: "none" sent alongside response_format:

{
  "model": "deepseek-v4-flash",
  "messages": [{ "role": "user", "content": "Classify and rewrite this ticket." }],
  "reasoning_effort": "none",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "triage",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "integer", "enum": [1, 2, 3, 4] },
          "rewrite": { "type": "string" }
        },
        "required": ["category", "rewrite"],
        "additionalProperties": false
      }
    }
  }
}

Every other live model enforces the same schema directly and needs no extra field. response_format: {"type": "text"} constrains nothing and is never refused. On /v1/responses the same remedy is a top-level reasoning_effort beside text.format. Working snippets are in Structured output.

An image, audio, or video content part

Accepted input is a per-model fact, stated on each model page as the Accepted input row. Qwen3.8 27B accepts image_url and input_image content parts as inline data URLs, up to 8 images per request: {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}} with png, jpeg, webp or gif. Remote image URLs are not fetched; download the image, base64-encode it, and send the data URL. Image tokens are counted by the server inside prompt_tokens and bill at the model’s input rate.On every other live model, and for audio and video everywhere, a non-text content part inside messages returns 400 with param: "messages" and code: "hosted_parameter_not_supported", naming the model and the part to remove. The part is refused rather than stripped, so a request carrying media is never billed as if it were text.

A prompt-cache control field

prompt_cache_key, prompt_cache_options and prompt_cache_retention are rejected with 400 hosted_parameter_not_supported. Prefix caching, where a model has it, is automatic server behavior and is not controllable per request.

Calling a paused model

A hosted model can be temporarily paused. The request returns 503 with code hosted_model_paused, error.paused_until carrying the scheduled return, and a Retry-After header in seconds capped at 60. Nothing is charged. The model id stays valid and stays listed by GET /v1/models with availability: "paused", so retry rather than re-resolving your configuration.

Streaming

Read deltas and ask for the usage frame.

Rate limits

The three layers that can refuse you, and how to back off.

Errors

Every status, code, and caller action.

Start here

Guides

API reference

What each model supports

Give reasoning models room

Turning reasoning off

What the gateway does with a field you send

Output limits

Three things a request can be refused for

Calling a paused model

Streaming

Rate limits

Errors

​What each model supports

​Give reasoning models room

​Turning reasoning off

​What the gateway does with a field you send

​Output limits

​Three things a request can be refused for

​Calling a paused model

​Related

Streaming

Rate limits

Errors

What each model supports

Give reasoning models room

Turning reasoning off

What the gateway does with a field you send

Output limits

Three things a request can be refused for

Calling a paused model

Related