Tag: attention
All the articles with the tag "attention".
Attention Alternatives: Sparse, Linear, and Recurrent
Updated:Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.
FlashAttention: Exact Attention at the Speed of Memory
Updated:Standard attention is slow at long contexts not because of its arithmetic but because it writes an n-by-n matrix of scores to memory and reads it back. FlashAttention computes exactly the same result without ever storing that matrix, using tiling and a softmax computed one block at a time. The online softmax derived and checked to within 1e-15, the memory traffic counted, a fused kernel measured against the three-step version, causal masking, and what FlashAttention 2 and 3, and Flash-Decoding, changed.
The KV Cache: How Transformers Avoid Recomputing the Past
Updated:Generating text recomputes every earlier token's keys and values unless they are kept. The KV cache keeps them, and it is exact: in Qwen2.5-1.5B-Instruct the next token matched at 32 of 32 steps. It turned a step at 4,096 tokens from 4.7 s into 65 ms on a laptop. Then what it costs: 28 KiB per token for Qwen, 800 KiB for Llama 2 13B, why decoding is limited by memory reads, how batch size trades latency for throughput, and how GQA, MLA and cross-layer sharing shrink the cache.
Sequences Before Attention: RNNs, Vanishing Gradients, and Why Transformers Won
Updated:A recurrent network compresses everything it has read into one fixed-size vector, updated one step at a time. Worked by hand and backpropagated through time, then the two problems that sentence contains: gradients that vanish over distance (measured for RNN, GRU and LSTM), and a fixed bottleneck that breaks sequence-to-sequence models (measured on a reversal task, with and without attention). Plus a character-level RNN trained on this blog, the baseline the tiny GPT trained later in the series has to beat.