Tag: linear-attention

All the articles with the tag "linear-attention".

• Attention Alternatives: Sparse, Linear, and Recurrent

  Updated:19 Oct, 2026

  Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.