Memory
Multi-query attention gives all query heads one shared key head and one shared value head. It is the most aggressive key-value sharing point in this attention family.
A single key-value stream minimizes cache size and related memory traffic. The tradeoff is architectural, and model quality depends on how the model was trained or converted.
Serving kernels broadcast the shared keys and values across query heads during attention. A checkpoint must already use compatible projections because this is not a scheduler-only setting.
Multi-query attention gives all query heads one shared key head and one shared value head. It is the most aggressive key-value sharing point in this attention family. Serving kernels broadcast the shared keys and values across query heads during attention. A checkpoint must already use compatible projections because this is not a scheduler-only setting.
A single key-value stream minimizes cache size and related memory traffic. The tradeoff is architectural, and model quality depends on how the model was trained or converted.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
