Idempotent retries

Send an Idempotency-Key header when a lost response or a network failure might make you retry. It is optional, it works on streaming and non-streaming requests alike, and it guarantees one generation and one charge. The same header and replay rules apply to /v1/messages. Use a non-blank ASCII value of 255 characters or fewer. One fresh key per logical request, reused only when you are retrying that same request. A UUID generated at the call site is a convenient source.

response = client.chat.completions.create(
    model="nemotron-3-5-lightning-30b",
    messages=[{"role": "user", "content": "Write one sentence."}],
    max_tokens=64,
    extra_headers={"Idempotency-Key": "YOUR_STABLE_IDEMPOTENCY_KEY"},
)

const response = await client.chat.completions.create(
  {
    model: "nemotron-3-5-lightning-30b",
    messages: [{ role: "user", content: "Write one sentence." }],
    max_tokens: 64,
  },
  { headers: { "Idempotency-Key": "YOUR_STABLE_IDEMPOTENCY_KEY" } },
);

curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: YOUR_STABLE_IDEMPOTENCY_KEY" \
  -d '{"model":"nemotron-3-5-lightning-30b","messages":[{"role":"user","content":"Write one sentence."}],"max_tokens":64}'

What a reused key does

Reuse the same key only with the same HTTP method, route, and request body. A key sent with a different body is a different logical request, and it is refused with 400 idempotency_mismatch rather than answered with the wrong reply. If you would rather opt out entirely, omit the header: a request without an Idempotency-Key runs with no deduplication at all.

Retrying a stream

A stream is not replayed byte for byte. We do not store the tokens we deliver, so there is nothing to send again, and a live event stream is never stored for replay. What the key protects on a streaming request is the part that costs money.

Situation	What you get
The original stream is still open	`409 idempotency_conflict` with `Retry-After: 1`. No second generation starts.
The original stream finished and settled	`200` with `X-RunInfra-Idempotent-Replay: true` and a non-streaming JSON receipt. Nothing is regenerated and nothing is billed again.
The original delivered nothing and was billed nothing	The key is released, so a retry runs normally, exactly as a request without a key would.

The receipt is an ordinary chat completion envelope, so an OpenAI-compatible client can parse it. content and finish_reason are null because the delivered text was never stored, and the replay facts ride in idempotent_replay.

{
  "id": "chatcmpl-8f0a1c2e-...",
  "object": "chat.completion",
  "created": 1786800000,
  "model": "qwen3-8-27b",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": null },
      "finish_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 1000,
    "completion_tokens": 500,
    "total_tokens": 1500,
    "prompt_tokens_details": { "cached_tokens": 200 },
    "cost": 0.00024,
    "runinfra": { "cost_microcents": 24000, "cached_input_tokens": 200 }
  },
  "idempotent_replay": {
    "replayed": true,
    "streamed": true,
    "content_replayed": false,
    "regenerated": false,
    "billed_again": false,
    "original_status": 200,
    "stream_outcome": "completed",
    "usage_basis": "provider_reported",
    "cost_microcents": 24000
  }
}

X-RunInfra-Idempotent-Replay is the signal on every route: read it before handing a response to a streaming parser. The idempotent_replay object is returned by /v1/chat/completions; on /v1/responses the reply is converted to a Responses object and only the header survives. If you need the generated text after a dropped stream, send a new request with a new key. That is new paid inference. Reusing the old key returns the receipt, not the text.

Only one charge per key

A retried request is charged once, streamed or not: a duplicate settlement for the same key is refused rather than billed twice. The key holds for the whole time a request can run, so a retry sent while a slow non-streaming request is still generating gets 409 idempotency_conflict and never starts a second generation. When a key cannot be checked, a keyed request is refused rather than run without protection. Every endpoint, chat completions, responses and messages included, returns 503 idempotency_unavailable with Retry-After before any inference starts or any charge is made; retry with the same key. On /v1/messages the same refusal arrives as Anthropic’s overloaded_error, which the Anthropic SDKs retry. A request without an Idempotency-Key is never affected.

Client request ids

X-Client-Request-Id is an optional correlation header. The gateway echoes a valid value back on the response. It is not a substitute for Idempotency-Key. Use a non-blank ASCII value of 512 characters or fewer.

Streaming

The frame shapes a retried stream is not replayed as.

Data retention

The 24 hour replay window is the one place a response body is stored.

Start here

Guides

API reference

Optimization

Help

Idempotent retries

What a reused key does

Retrying a stream

Only one charge per key

Client request ids

Streaming

Data retention

​What a reused key does

​Retrying a stream

​Only one charge per key

​Client request ids

​Related

Streaming

Data retention

What a reused key does

Retrying a stream

Only one charge per key

Client request ids

Related