Memory
Prefix caching preserves KV cache blocks for prompt prefixes that recur across requests. A matching request can reuse the cached prefix and compute only the uncached suffix.
Reuse removes repeated prefill work for shared instructions, templates, or documents. It can reduce first-token latency and compute cost when cache hits are frequent and prefixes are long enough.
The server identifies reusable prefixes, stores their cache blocks, and checks later requests for exact matches. Gains depend on prompt repetition, cache capacity, eviction behavior, and the cost of lookup.
Prefix caching preserves KV cache blocks for prompt prefixes that recur across requests. A matching request can reuse the cached prefix and compute only the uncached suffix. The server identifies reusable prefixes, stores their cache blocks, and checks later requests for exact matches. Gains depend on prompt repetition, cache capacity, eviction behavior, and the cost of lookup.
Reuse removes repeated prefill work for shared instructions, templates, or documents. It can reduce first-token latency and compute cost when cache hits are frequent and prefixes are long enough.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
