Streaming responses

On a model whose page lists streaming, stream tokens as they are generated so a user sees the answer appear instead of waiting for all of it. Any chat UI wants this. So does anything long enough that silence looks like a hang.

Minimal code

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)

stream = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Tell me a short story"}],
    max_tokens=16384,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});

const stream = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [{ role: "user", content: "Tell me a short story" }],
  max_tokens: 16384,
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content ?? "");
}

curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [{"role":"user","content":"Tell me a short story"}],
    "max_tokens": 16384,
    "stream": true
  }'

What your loop will see

What to tune

Parameter	Effect
`stream_options.include_usage`	Adds the final usage frame with token counts. Off by default.
`max_tokens`	Hard cap on generated length. Reasoning models spend part of this budget before answering, so keep it generous.
`temperature`	Higher is more surprise per token.
`stop`	Up to four stop sequences. The stream ends early on a match.

Common mistakes

Forgetting flush=True in Python. Without it stdout buffers and the tokens arrive in clumps.
Treating an error frame as a network failure. A stream that dies upstream ends with one final frame carrying the standard error envelope, on the same HTTP 200. Handle it inside your frame loop.
Breaking the loop too early. The final content frame carries finish_reason and an empty delta.content. Keep reading until the stream closes.
Streaming through a buffering proxy. Some edges buffer server-sent events. Stream direct, or disable buffering on that layer.
Mixing n > 1 with streaming. Allowed, but every delta carries its own choices[].index. Route by index or the outputs interleave.

Next steps

Stream reference

Exact frame shapes, and the usage frame.

Tool calling

Let the model act, then answer.

Structured output

Get JSON that matches your schema.

Start here

Guides

API reference

Streaming responses

Minimal code

What your loop will see

What to tune

Common mistakes

Next steps

Stream reference

Tool calling

Structured output

​Minimal code

​What your loop will see

​What to tune

​Common mistakes

​Next steps

Stream reference

Tool calling

Structured output

Minimal code

What your loop will see

What to tune

Common mistakes

Next steps