Streaming

For a model whose page lists streaming, set stream: true on POST /v1/chat/completions to get a Server-Sent Events response. Add stream_options.include_usage when your client needs the usage frame. /v1/messages uses a different, event-named stream grammar described in Anthropic Messages.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
stream = client.chat.completions.create(
    model="nemotron-3-5-lightning-30b",
    messages=[{"role": "user", "content": "Count from one to three."}],
    max_tokens=64,
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.usage:
        print(f"\nusage: {chunk.usage}")
        continue
    print(chunk.choices[0].delta.content or "", end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const stream = await client.chat.completions.create({
  model: "nemotron-3-5-lightning-30b",
  messages: [{ role: "user", content: "Count from one to three." }],
  max_tokens: 64,
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  if (chunk.usage) console.error("usage:", chunk.usage);
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"nemotron-3-5-lightning-30b","messages":[{"role":"user","content":"Count from one to three."}],"max_tokens":64,"stream":true,"stream_options":{"include_usage":true}}'

What arrives, and when

Frame shapes

The response carries Content-Type: text/event-stream, Cache-Control: no-cache and Connection: keep-alive. Content arrives in OpenAI-compatible data: frames, and partial output is on choices[].delta.

data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"}}]}

The usage frame has an empty choices array. On a hosted model it also carries the request’s cost and the count of input tokens billed at the cached rate, both described in Chat completions:

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16,"prompt_tokens_details":{"cached_tokens":0},"cost":0.0000028,"runinfra":{"cost_microcents":280,"cached_input_tokens":0}}}

A completed stream ends with data: [DONE].

data: [DONE]

A stream the gateway terminates early ends with an error frame and no [DONE]. A transport interruption can close it without an error frame. Treat a missing [DONE] as an incomplete response even when the HTTP status was 200. Retrying with the same Idempotency-Key returns the original request’s receipt, or 409 while it is still running; it never replays the text. To get an answer, send a new request with a new key, which is new paid inference. See Retrying a stream.

data: {"error":{"message":"The upstream stream ended before signalling completion. Please retry.","type":"api_error","param":null,"code":"upstream_stream_error"},"request_id":"req_example_0000"}

code is upstream_error when the model stream ended, hosted_inference_response_timeout when the response deadline did, and upstream_stream_error otherwise. See Errors.

Is usage always included?

No. The gateway always asks the model for usage so it can settle the request, but it forwards the usage frame to you only when your request carried stream_options.include_usage: true. Omit it and content frames still stream while usage-only frames stay hidden. A usage frame is never synthesized: if the model reports no usage for a request, none is sent. After an interrupted request, recovery uses the usage evidence that was persisted. A failed checkpoint write can leave that evidence missing or stale, so neither the delivered token count nor the final charge can be inferred from the last content chunk. Keep the request id for support. A new key starts new paid inference and cannot recover the old response. There is one exception, so you can account for generated tokens even when no answer was delivered. When every choice finishes without final answer content, the gateway forwards one annotated usage frame whether or not you opted in.

When a stream ends without an answer

A choice that reaches its generation limit before producing answer content carries a machine-readable status on its terminal frame:

data: {"choices":[{"index":0,"delta":{},"finish_reason":"length","runinfra":{"output_status":{"code":"generation_limit_reached_before_answer","message":"The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."}}}]}

If another terminal finish_reason produced no answer, the code is no_answer_content instead. Inspect the preserved finish_reason before deciding whether to retry. The annotated usage frame then reports how much of the generated output was not an answer:

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":64,"total_tokens":76,"prompt_tokens_details":{"cached_tokens":0},"cost":0,"runinfra":{"output_token_accounting":{"visible_answer_tokens":0,"non_answer_completion_tokens":64,"sources":{"visible_answer_tokens":"proxy_classified","non_answer_completion_tokens":"provider_reported"}},"cost_microcents":0,"cached_input_tokens":0}}}

A stream that produced no answer settles at zero, so this frame says cost: 0 and cached_tokens: 0 whatever the model reported: nothing was priced, at either rate. The usual cause is an output budget too small for a model that reasons first. See give reasoning models room.

The same Idempotency-Key reads the original request’s receipt; it does not replay delivered tokens. Getting a new answer requires a new request and a new key, which starts new paid inference. Read Retrying a stream before retrying a stream.

Chat completions

The request contract every stream starts from.

Idempotent retries

Retry a dropped stream without paying twice.

Errors

Every status, code, and caller action.

Start here

Coding agents

Guides

API reference

What arrives, and when

Frame shapes

Is usage always included?

When a stream ends without an answer

Chat completions

Idempotent retries

Errors

​What arrives, and when

​Frame shapes

​Is usage always included?

​When a stream ends without an answer

​Related

Chat completions

Idempotent retries

Errors

What arrives, and when

Frame shapes

Is usage always included?

When a stream ends without an answer

Related