What does Attention sinks mean in inference serving?
Attention sinks are the tendency for attention to concentrate on initial tokens even when those tokens have little semantic importance. StreamingLLM retains their cached keys and values alongside a sliding window to preserve generation quality. Some architectures, including gpt-oss, implement a learned per-head bias in the softmax denominator as a separate sink mechanism. Serving kernels must preserve that bias to compute the model as trained.