Memory
Paged attention stores each sequence's KV cache in fixed-size blocks rather than one contiguous allocation. A block table maps logical token positions to physical cache blocks during attention.
Block allocation reduces memory fragmentation and lets the server reuse freed capacity between sequences. Better cache occupancy can support larger active batches before memory becomes the limiting resource.
The runtime allocates blocks as sequences grow, then returns blocks when sequences finish or caches expire. Block size and eviction policy trade metadata work against wasted memory and cache reuse.
Paged attention stores each sequence's KV cache in fixed-size blocks rather than one contiguous allocation. A block table maps logical token positions to physical cache blocks during attention. The runtime allocates blocks as sequences grow, then returns blocks when sequences finish or caches expire. Block size and eviction policy trade metadata work against wasted memory and cache reuse.
Block allocation reduces memory fragmentation and lets the server reuse freed capacity between sequences. Better cache occupancy can support larger active batches before memory becomes the limiting resource.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
