Skip to main content
Cached input is billed at the cached rate, roughly a tenth of the standard input price on most models. For an agent that sends a growing conversation on every turn, that is the difference between paying for your whole context once and paying for it on every request. Caching is automatic. There is no header to set and no parameter to send. What you control is whether your prompt stays reusable between turns.

​
The rule

The cache matches a prefix: the longest run of tokens from the start of your prompt that is byte-identical to a previous request. Everything from the first difference onward is recomputed and billed as uncached. So an agent turn that appends to the end of its context reuses almost everything. A turn that changes something near the beginning reuses almost nothing, however small the change.

​
What breaks a prefix

Every item below has been observed reducing a real workload’s hit rate. They are ordered by how often they turn out to be the cause.
The single most common cause. Current time: 2026-08-21T09:14:22Z at the top of a system prompt changes on every request, so nothing after it can ever match.Move it to the end of the conversation, as the last user message or a trailing system note. It is just as visible to the model there and it stops invalidating everything behind it.
If your tool list is built from a set, a dictionary, or a directory listing, its order can change between processes even when the tools do not. Sort tool definitions by name before serializing, once, and keep that order for the life of the session.
JSON.stringify over an object preserves insertion order, which can differ between runs. Python’s json.dumps accepts sort_keys=True. If you build tool schemas or context blocks programmatically, serialize them deterministically.Whitespace counts too: a pretty-printer that changes indentation between versions changes the bytes.
Anything unique per request belongs outside the prompt. If you need the model to know a session id, put it at the end, not in the system block.
Summarizing older turns is good practice, but replacing the middle of a conversation invalidates the prefix from the summary onward. Compact at a boundary you keep stable, and prefer appending a summary to rewriting history in place.
If retrieved context is sorted by a score that shifts slightly between calls, the block order changes. Sort retrieved chunks by a stable key, such as document id, rather than by score.

​
Reading your own hit rate

GET /api/usage/cache reports, per API key and per model, how many input tokens were served from cache and what that saved. A low rate on a workload that should be repetitive means something above is changing. The same response includes a distribution, not just an average. Its most useful field is the mode: the cached-token count that recurs most often across your requests. Because the serving engine caches in fixed blocks, that number is where your prompt stops matching itself. A worked example from real traffic: one workload showed a mode of 9,984 cached tokens on a fifth of its requests, against prompts averaging about 70,000 tokens. 9,984 is exactly 39 blocks. Everything up to roughly token 10,000 was reusable and nothing after it was, on every affected turn. That is the signature of a field changing early in the prompt, not of a cache problem. A sibling workload on the same model reused 92 percent of a 465,000-token prompt. If your mode sits far below your prompt size and recurs, look at what is between that offset and the start of your context.

​
Keeping a session warm

A prefix stays cached for a bounded idle period, published per model as session_affinity_idle_ttl_seconds on /v1/models. Turns that arrive within that window reuse it. Retention is best effort: capacity pressure can evict an idle prefix sooner, and no model guarantees a lifetime. If your agent pauses between turns to run tools, that pause is normal and the cache is designed for it. You do not need to send keepalive traffic.

​
What does not affect caching

  • Sampling parameters. temperature, top_p, seed and max_tokens are not part of the prefix.
  • Streaming. A streamed and a non-streamed request cache identically.
  • Your traffic volume. Caching is per prefix, not a tier or an allowance.

​
Isolation

On models that publish cache_isolation: "isolated" in /v1/models, a cached prefix is scoped to your workspace. Another customer sending the same bytes does not read yours, and you do not read theirs. Models that do not publish it do not report cached counts and are billed at the standard input rate.