Memory
Flash attention is an IO-aware exact attention method that tiles computation around limited on-chip memory. It avoids materializing the full attention score matrix in high-bandwidth memory.
Fewer transfers between device memory and on-chip storage reduce attention memory traffic. The benefit depends on sequence shape, supported kernels, and the surrounding model workload.
A compatible kernel processes query, key, and value tiles while maintaining numerically stable running statistics. Kernel selection and tile sizes vary with data type, head shape, and accelerator.
Flash attention is an IO-aware exact attention method that tiles computation around limited on-chip memory. It avoids materializing the full attention score matrix in high-bandwidth memory. A compatible kernel processes query, key, and value tiles while maintaining numerically stable running statistics. Kernel selection and tile sizes vary with data type, head shape, and accelerator.
Fewer transfers between device memory and on-chip storage reduce attention memory traffic. The benefit depends on sequence shape, supported kernels, and the surrounding model workload.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
