> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt caching for agents

> Cached input costs about a tenth of uncached. Here is what keeps a prefix reusable across an agent's turns, and what quietly breaks it.

Cached input is billed at the cached rate, roughly a tenth of the standard
input price on most models. For an agent that sends a growing conversation on
every turn, that is the difference between paying for your whole context once
and paying for it on every request.

Caching is automatic. There is no header to set and no parameter to send. What
you control is whether your prompt stays *reusable* between turns.

## The rule

The cache matches a **prefix**: the longest run of tokens from the start of
your prompt that is byte-identical to a previous request. Everything from the
first difference onward is recomputed and billed as uncached.

So an agent turn that appends to the end of its context reuses almost
everything. A turn that changes something near the beginning reuses almost
nothing, however small the change.

```
Turn 1:  [system][tools][file context][user msg 1]
Turn 2:  [system][tools][file context][user msg 1][assistant][user msg 2]
                                                  ^ cache matches to here
```

## What breaks a prefix

Every item below has been observed reducing a real workload's hit rate. They
are ordered by how often they turn out to be the cause.

<AccordionGroup>
  <Accordion title="A timestamp or date in the system prompt">
    The single most common cause. `Current time: 2026-08-21T09:14:22Z` at the top
    of a system prompt changes on every request, so nothing after it can ever
    match.

    Move it to the **end** of the conversation, as the last user message or a
    trailing system note. It is just as visible to the model there and it stops
    invalidating everything behind it.
  </Accordion>

  <Accordion title="Tool definitions in an unstable order">
    If your tool list is built from a set, a dictionary, or a directory listing,
    its order can change between processes even when the tools do not. Sort tool
    definitions by name before serializing, once, and keep that order for the life
    of the session.
  </Accordion>

  <Accordion title="Non-deterministic JSON serialization">
    `JSON.stringify` over an object preserves insertion order, which can differ
    between runs. Python's `json.dumps` accepts `sort_keys=True`. If you build
    tool schemas or context blocks programmatically, serialize them
    deterministically.

    Whitespace counts too: a pretty-printer that changes indentation between
    versions changes the bytes.
  </Accordion>

  <Accordion title="A session or request id inside the prompt">
    Anything unique per request belongs outside the prompt. If you need the model
    to know a session id, put it at the end, not in the system block.
  </Accordion>

  <Accordion title="Context compaction that rewrites the middle">
    Summarizing older turns is good practice, but replacing the middle of a
    conversation invalidates the prefix from the summary onward. Compact at a
    boundary you keep stable, and prefer appending a summary to rewriting history
    in place.
  </Accordion>

  <Accordion title="Reordering retrieved documents">
    If retrieved context is sorted by a score that shifts slightly between calls,
    the block order changes. Sort retrieved chunks by a stable key, such as
    document id, rather than by score.
  </Accordion>
</AccordionGroup>

## Reading your own hit rate

`GET /api/usage/cache` reports, per API key and per model, how many input
tokens were served from cache and what that saved. A low rate on a workload
that should be repetitive means something above is changing.

The same response includes a **distribution**, not just an average. Its most
useful field is the mode: the cached-token count that recurs most often across
your requests. Because the serving engine caches in fixed blocks, that number
is where your prompt stops matching itself.

A worked example from real traffic: one workload showed a mode of **9,984
cached tokens on a fifth of its requests**, against prompts averaging about
70,000 tokens. 9,984 is exactly 39 blocks. Everything up to roughly token
10,000 was reusable and nothing after it was, on every affected turn. That is
the signature of a field changing early in the prompt, not of a cache problem.
A sibling workload on the same model reused 92 percent of a 465,000-token
prompt.

If your mode sits far below your prompt size and recurs, look at what is
between that offset and the start of your context.

## Keeping a session warm

A prefix stays cached for a bounded idle period, published per model as
`session_affinity_idle_ttl_seconds` on [`/v1/models`](/docs/api-reference/models).
Turns that arrive within that window reuse it. Retention is best effort:
capacity pressure can evict an idle prefix sooner, and no model guarantees a
lifetime.

If your agent pauses between turns to run tools, that pause is normal and the
cache is designed for it. You do not need to send keepalive traffic.

## What does not affect caching

* **Sampling parameters.** `temperature`, `top_p`, `seed` and `max_tokens` are
  not part of the prefix.
* **Streaming.** A streamed and a non-streamed request cache identically.
* **Your traffic volume.** Caching is per prefix, not a tier or an allowance.

## Isolation

On models that publish `cache_isolation: "isolated"` in
[`/v1/models`](/docs/api-reference/models), a cached prefix is scoped to your
workspace. Another customer sending the same bytes does not read yours, and
you do not read theirs. Models that do not publish it do not report cached
counts and are billed at the standard input rate.