Streaming Model APIs responses

Set stream: true to receive a Server-Sent Events response from POST /v1/chat/completions.

Stream a response

Set stream_options.include_usage to true when your client needs the usage-only chunk.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
stream = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Count from one to three."}],
    max_tokens=64,
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.usage:
        print(f"\nusage: {chunk.usage}")
        continue
    print(chunk.choices[0].delta.content or "", end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const stream = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [{ role: "user", content: "Count from one to three." }],
  max_tokens: 64,
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  if (chunk.usage) console.error("usage:", chunk.usage);
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Count from one to three."}],"max_tokens":64,"stream":true,"stream_options":{"include_usage":true}}'

Event shape

The response uses these headers:

Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

Content arrives in OpenAI-compatible data: events. Read partial output from choices[].delta.

data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"}}]}

When client-visible usage is enabled, the usage event has an empty choices array:

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16}}

The gateway forwards data: [DONE] when the upstream stream sends it.

data: [DONE]

Is usage always included?

No. The gateway always requests usage from the hosted model so it can settle the request. It forwards the usage-only event to your client only when your original request includes:

{
  "stream_options": {
    "include_usage": true
  }
}

If you omit that option, content chunks still stream, but the usage-only chunk is hidden from your client. If the upstream stream omits usage, the gateway does not synthesize a client-visible usage chunk.

Streaming requests do not use the chat idempotency cache. See Idempotent retries before retrying a stream.

Getting started

Endpoints

Reference

Streaming Model APIs responses

Stream a response

Event shape

Is usage always included?

​Stream a response

​Event shape

​Is usage always included?

Stream a response

Event shape

Is usage always included?