Agent inference

Session affinity

What it is

Source: RunInfra, Session affinity, read 2026-09-25

Source: OpenAI, Prompt caching and routing context, read 2026-09-25

Why it moves cost and latency

Keeping repeated agent context near a warm cache can lower time to first token and cached input cost. Affinity does not guarantee a cache hit or control cache retention, and prompt_cache_key groups requests without holding an established session in place.

What it looks like in practice

On RunInfra, keep x-session-id stable across turns, or use x-session-affinity; a fan-out worker follows x-parent-session-id only when it names no session of its own. The gateway also accepts runinfra.sessionId in the body and x-runinfra-session, with prompt_cache_key as a fallback identity. Claude Code, Codex CLI, and OpenCode send their own session headers with no configuration: RunInfra reads Claude Code's x-claude-code-session-id and subagent x-claude-code-agent-id, and Codex CLI's thread-id or session-id when x-codex-window-id is present.

Related terms

Questions this definition answers

What does Session affinity mean in inference serving?

Session affinity is a routing policy that keeps requests from the same conversation together to improve reuse of cached prompt work. A stable session identity lets a gateway recognize later turns and route them with the session's cache in mind. On RunInfra, keep x-session-id stable across turns, or use x-session-affinity; a fan-out worker follows x-parent-session-id only when it names no session of its own. The gateway also accepts runinfra.sessionId in the body and x-runinfra-session, with prompt_cache_key as a fallback identity. Claude Code, Codex CLI, and OpenCode send their own session headers with no configuration: RunInfra reads Claude Code's x-claude-code-session-id and subagent x-claude-code-agent-id, and Codex CLI's thread-id or session-id when x-codex-window-id is present.