Memory
Attention sinks are retained early tokens that receive attention after later tokens move through a sliding window. Keeping those tokens can stabilize generation for models that exhibit this behavior.
Dropping every old token can shift attention behavior and degrade streaming quality. Sink retention uses a small fixed cache region but does not create unlimited context memory.
The runtime preserves selected initial positions while evicting older non-sink positions. The retained count and its effectiveness vary by model and require validation.
Attention sinks are retained early tokens that receive attention after later tokens move through a sliding window. Keeping those tokens can stabilize generation for models that exhibit this behavior. The runtime preserves selected initial positions while evicting older non-sink positions. The retained count and its effectiveness vary by model and require validation.
Dropping every old token can shift attention behavior and degrade streaming quality. Sink retention uses a small fixed cache region but does not create unlimited context memory.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
