Tag: online-softmax
All the articles with the tag "online-softmax".
FlashAttention: Exact Attention at the Speed of Memory
Updated:Standard attention is slow at long contexts not because of its arithmetic but because it writes an n-by-n matrix of scores to memory and reads it back. FlashAttention computes exactly the same result without ever storing that matrix, using tiling and a softmax computed one block at a time. The online softmax derived and checked to within 1e-15, the memory traffic counted, a fused kernel measured against the three-step version, causal masking, and what FlashAttention 2 and 3, and Flash-Decoding, changed.