Tag: sparse-attention
All the articles with the tag "sparse-attention".
Attention Alternatives: Sparse, Linear, and Recurrent
Updated:Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.
Anatomy of a Frontier Model: Reading DeepSeek-V4 Through Part V
Updated:DeepSeek-V4 (April 2026) is two mixture-of-experts models with a million-token context: 1.6 trillion parameters with 49 billion active, and 284 billion with 13 billion active. Read through Part V, each unfamiliar part answers a cost the earlier posts measured: compressed and sparse attention for the KV cache, an indexer for the sparse selection, a widened residual stream kept stable by a doubly stochastic constraint, the Muon optimizer, FP4 experts, and on-policy distillation from more than ten teachers. The KV cache recomputed from the configuration: 2% of a standard model's at a million tokens.