What does Cached input tokens mean in inference serving?
Cached input tokens are prompt tokens whose stored computation is reused from a matching prefix instead of being recomputed. RunInfra matches a byte-identical prefix, and everything from the first difference onward is recomputed and billed as uncached. Each protocol reports cached input in its own field: Chat Completions in usage.prompt_tokens_details.cached_tokens and usage.runinfra.cached_input_tokens, which report zero rather than being absent when nothing was billed as cached; the Responses API in usage.input_tokens_details.cached_tokens; Anthropic Messages in usage.cache_read_input_tokens. Keep the leading prompt stable by removing changing timestamps and per-request IDs, stabilizing tool ordering and JSON serialization, and avoiding mid-conversation rewrites and reordered retrieved documents. Reuse is counted in whole cache blocks, so a trailing partial block is billed at the input rate even when its text matches.