Rate limits

Three independent layers can refuse a Model APIs request, and they run in this order:

A connection gate, before your credential is read.
A per-key request window, requests per minute for that key.
Hosted admission, concurrency and workspace tokens per minute on a shared model.

All three answer HTTP 429. The per-key and hosted layers use the standard error envelope, with different error.code values and different headers.

Read the header, never a constant

Every 429 from every layer carries both retry headers.

Retry-After: <seconds>
Retry-After-Ms: <milliseconds>

Retry-After-Ms is the same instant in milliseconds. Read it first if your client supports it, and the OpenAI and Anthropic SDKs both do: it is the only one of the two that can express a wait shorter than a second without rounding up. Retry-After is the RFC 9110 header in whole seconds, always at least 1, always rounded up, so it never advises you to retry early.

On a hosted admission refusal these values are an estimate measured from the limiter that refused you, not a promise. A concurrency refusal clears when some in-flight request finishes, which no gateway can predict, so the number is derived from how that limiter is behaving: how long it has gone without admitting anything, and when its oldest slot expires. It moves between refusals and is capped at 8 seconds, so a busy moment can never park your client. On the per-key window and on a paused model the value is authoritative rather than estimated, because those clear at a known time, and it is not capped there.

Layer 1, the connection gate

Before the API reads your key, a gate bounds how fast unrecognized credentials can be looked up, so a flood of invalid keys cannot crowd out real traffic. It applies per source address and across the service as a whole. There is no warm-up and nothing to request: a new key works on its first call. What is bounded is the rate at which credentials that have never authenticated can be presented, and once a key succeeds once it stops counting against that budget. Steady traffic from a working key never meets this gate. You are most likely to see it when starting many workers at once with a brand new key, or when a script is retrying a key that is simply wrong. Rolling a new key out on one request before fanning out avoids it entirely. The budgets are not published, because this is a defensive control and it is tuned. If the gate cannot reach the state it needs, it fails closed with 503 and a short Retry-After rather than admitting unbounded work.

Layer 2, the per-key request window

A rolling 60 second window. A key with no custom limit follows the workspace’s current default, so a workspace upgrade applies without rotating the key. A custom per-key limit stays in place, clamped to the workspace maximum.

{
  "error": {
    "message": "Rate limit exceeded. Please retry after the reset time.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  },
  "request_id": "req_example_1000"
}

Header	Meaning
`X-RateLimit-Limit`	The key’s request count for the 60 second window.
`X-RateLimit-Remaining`	Requests left in the current window. It is `0` on this refusal.
`X-RateLimit-Reset`	Unix seconds when the window resets.
`Retry-After`	Seconds to wait.

The window slides, so capacity returns gradually as individual requests age past 60 seconds. There is no clock edge where the whole budget refills at once, which is why pacing evenly beats bursting. If the limiter itself cannot be reached, the gateway fails closed with 503 limiter_unavailable rather than serving unmetered.

Layer 3, hosted admission

Four gates, checked in that order. A hosted refusal carries limit inside error, plus headers naming the gate that stopped you.

X-Hosted-Limit-Scope: <scope>
X-Hosted-Limit-Value: <limit>
Retry-After: <seconds>
Retry-After-Ms: <milliseconds>

`error.code`	`X-Hosted-Limit-Scope`	What clears it
`hosted_per_key_concurrency_limit`	`api-key-concurrency`	An in-flight request using the same key finishes.
`hosted_workspace_concurrency_limit`	`workspace-concurrency`	Any in-flight hosted request in the workspace finishes.
`hosted_saturated_large_prompt`	`shared-model-saturation`	Load drops below the pressure watermark, usually within seconds. Applies only to very large prompts while the model is near its concurrency limit; it is not a size limit and the same request admits unchanged when load drops.
`hosted_shared_model_concurrency_limit`	`shared-model-concurrency`	Shared capacity on that model frees up.
`hosted_workspace_tpm_limit`	`workspace-tokens-per-minute`	The token window rolls forward, or you send fewer tokens.

Every one of the four uses the same envelope. This is the API-key concurrency case, and the numbers in it are an example:

{
  "error": {
    "limit": 16,
    "message": "This API key reached its limit of 16 concurrent hosted requests. Wait for an in-flight request to finish or reduce this key's concurrency.",
    "type": "rate_limit_error",
    "param": null,
    "code": "hosted_per_key_concurrency_limit"
  },
  "request_id": "req_example_1001"
}

Always read error.limit and X-Hosted-Limit-Value from the response you got, never a number from this page. They change with capacity.

Handling a 429 in code

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    max_retries=5,
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Health check"}],
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
  maxRetries: 5,
});

const response = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [{ role: "user", content: "Health check" }],
});
console.log(response.choices[0]?.message?.content);

An SDK with retries enabled does the right thing with no code from you, because every layer sends the headers. If you back off yourself, add jitter when many workers share a key so they do not all return at the same instant.

Three cases worth planning for

A 429 before any request of yours has succeeded, on a key you just created, is the connection gate rather than a quota. Retry once after Retry-After with a single request, then resume normal concurrency.
One request that alone exceeds the TPM limit is the case where waiting cannot help. It carries no timing estimate. Reduce the input or the maximum output tokens before retrying.
Adding API keys does not buy capacity. The workspace concurrency cap is derived from shared-model capacity, so an eight-slot shared model exposes two slots per workspace, and two workspaces can never occupy the whole node. Hosted token-rate admission uses the same request token estimate as billing, not a byte count.

Errors

Every status, code, and whether a retry can help.

Models

Discover the limits that bind your key at boot.

Start here

Guides

API reference

Read the header, never a constant

Layer 1, the connection gate

Layer 2, the per-key request window

Layer 3, hosted admission

Handling a 429 in code

Three cases worth planning for

Errors

Models

​Read the header, never a constant

​Layer 1, the connection gate

​Layer 2, the per-key request window

​Layer 3, hosted admission

​Handling a 429 in code

​Three cases worth planning for

​Related

Errors

Models

Read the header, never a constant

Layer 1, the connection gate

Layer 2, the per-key request window

Layer 3, hosted admission

Handling a 429 in code

Three cases worth planning for

Related