Skip to main content
Cached input is billed at the cached rate, roughly a tenth of the standard input price on most models. For an agent that sends a growing conversation on every turn, that is the difference between paying full price for your whole context once and about a tenth of it on every later turn. Caching is automatic. No header or parameter is required. What you control is whether your prompt stays reusable between turns.

​
The rule

The cache matches a prefix: the longest run of tokens from the start of your prompt that is byte-identical to a previous request. Everything from the first difference onward is recomputed and billed as uncached. So an agent turn that appends to the end of its context reuses almost everything. A turn that changes something near the beginning reuses almost nothing, however small the change.
What a prefix cache reusesthe match runs from the startTurn 1, first sendsystemtoolsfile contextuser 1Nothing to match yetTurn 2, appended at the endsystemtoolsfile contextuser 1assistantuser 2First differenceReused, cached rateRecomputedTurn 2, with a timestamp at the topsystem + timetoolsfile contextuser 1assistantuser 2Nothing reused, every token recomputedEverything from the first difference onward is recomputed and billed as uncached.Column widths are illustrative. Cached input is billed at about a tenth of standard input on most models.
What a prefix cache reusesthe match runs from the startTurn 1, first sendsystemtoolsfile contextuser 1Nothing to match yetTurn 2, appended at the endsystemtoolsfile contextuser 1assistantuser 2First differenceReused, cached rateRecomputedTurn 2, with a timestamp at the topsystem + timetoolsfile contextuser 1assistantuser 2Nothing reused, every token recomputedEverything from the first difference onward is recomputed and billed as uncached.Column widths are illustrative. Cached input is billed at about a tenth of standard input on most models.
The match is counted in whole cache blocks, and the trailing partial block is billed at the input rate. The unit size varies by model, so a very short prompt may never hit, and the discount does the most work on long prefixes. Read usage.prompt_tokens_details.cached_tokens on your own responses to see how much of your prompt is matching. One more shape matters on qwen3-8-flash-next: a conversation that appends warms on turn 2, as in the diagram, but a shared prefix re-sent with a changing suffix, many questions against one document, may take an extra send before cached_tokens shows it warm.

​
What breaks a prefix

Every item below has been observed reducing a real workload’s hit rate. They are ordered by how often they turn out to be the cause.
The single most common cause. Current time: 2026-08-21T09:14:22Z at the top of a system prompt changes on every request, so nothing after it can ever match.Move it to the end of the conversation, as the last user message or a trailing system note. It is just as visible to the model there and it stops invalidating everything behind it.
If your tool list is built from a set, a dictionary, or a directory listing, its order can change between processes even when the tools do not. Sort tool definitions by name before serializing, once, and keep that order for the life of the session.
JSON.stringify over an object preserves insertion order, which can differ between runs. Python’s json.dumps accepts sort_keys=True. If you build tool schemas or context blocks programmatically, serialize them deterministically.Whitespace counts too: a pretty-printer that changes indentation between versions changes the bytes.
Anything unique per request belongs outside the prompt. If you need the model to know a session id, put it at the end, not in the system block.
Summarizing older turns is good practice, but replacing the middle of a conversation invalidates the prefix from the summary onward. Compact at a boundary you keep stable, and prefer appending a summary to rewriting history in place.
If retrieved context is sorted by a score that shifts slightly between calls, the block order changes. Sort retrieved chunks by a stable key, such as document id, rather than by score.

​
Reading your own hit rate

Every hosted response reports usage.prompt_tokens_details.cached_tokens, the count of input tokens that request was billed at the cached input rate, and the same number as usage.runinfra.cached_input_tokens beside the cost. It is 0 rather than absent when nothing was billed as cached. Print it on every turn: on a workload that should be repetitive, the value that keeps coming back is where your prompt stops matching itself. The field is described in Chat completions. Your workspace’s cache report is on the dashboard’s Usage Analytics page, read with your signed-in dashboard session rather than an API key. The Cache by API key band carries the workspace totals: cached input tokens, the confirmed hit rate, confirmed misses, and confirmed savings. The By API key and By model tables on the same page carry cached tokens, hit rate, and savings per key and per model. A low rate on a workload that should be repetitive means something above is changing. To compare many requests, record their cached-token counts and look at the distribution, not just an average. Its most useful summary is the mode: the cached-token count that recurs most often. Because matching is counted in whole cache blocks, this is roughly where your prompt stops matching itself; past that point, a field in your prompt may be differing between requests.
A worked example from real traffic, measured in August 2026: one workload showed a mode of 9,984 cached tokens on a fifth of its requests, against prompts averaging about 70,000 tokens. Everything up to roughly token 10,000 was reusable and nothing after it was, on every affected turn. That is the signature of a field changing early in the prompt, not of a cache problem. A sibling workload on the same model reused 92 percent of a 465,000-token prompt. These are measured examples, not a promised hit rate. If the count that recurs sits far below your prompt size, look at what is between that offset and the start of your context.

​
Keeping a session warm

On every chat model but one, retention of a warm prefix is best effort: an idle prefix can be evicted under capacity pressure, and no lifetime is promised. The exception is GLM 5.3 Flash (glm-5-3-flash), where the prefix behind a session you name is kept across idle gaps, for up to 6 hours at the fastest reuse and up to 72 hours since the last request, at no charge. Those windows are ceilings rather than guarantees; idle state can be dropped earlier under capacity pressure, and the tier table is on the session affinity page. On /v1/models, cache_tiers and cache_retention tell you which kind of cache a model runs. best_effort with cache_tiers ["gpu"] is the GPU copy alone. tiered_host_memory means a prefix can outlive the fastest tier: DeepSeek V4 Pro (deepseek-v4-pro) publishes it, and retention there is still best effort. The glm-5-3-flash keep windows are published on the session affinity and data retention pages rather than as fields in that listing. session_affinity_idle_ttl_seconds is a routing window, how long a quiet session keeps its place close to its cache, not a retention promise. The cached-token count on the first turn after a pause is the readout of what survived it. If the copy was lost while your agent ran tools, waited on a human, or slept overnight, that turn is billed as a miss; the count and the bill agree. The matching rule at the top of this page is unchanged: a returning prompt must be byte-identical to the prefix it wants back. On any chat model, a stable x-session-id keeps a session’s follow-up turns close to its warm cache, and pins that placement for the model’s published session_affinity_idle_ttl_seconds idle window. Without one, a prompt_cache_key in the request body routes the request by its value but does not pin it. Without either, placement still keeps steady sessions warm; the full placement order is on the session affinity page.

​
What does not affect caching

  • Sampling parameters. temperature, top_p, seed, and max_tokens are not part of the prefix.
  • Streaming. A streamed and a non-streamed request cache identically.
  • Your traffic volume. Caching is per prefix, not a plan or an allowance.

​
Isolation

On models that publish cache_isolation: "isolated" in /v1/models, a cached prefix is scoped to your workspace within the model’s published isolation scope. Another customer does not reuse your isolated prefix, and you do not reuse theirs. That scoping applies on every request. Read the per-model tier scope in Data retention. On a model that does not publish isolation, a hit is neither disclosed nor priced. A billable response charges every input token at the standard input rate. The response reports cached_tokens: 0, including when it settles at zero.

​
Next steps

Session affinity

Keep every turn close to your warm prefix.

Models

Read cache isolation, tiers, and retention per model.

Tool calling

The agent loop these prefixes come from.