Tag: muon

All the articles with the tag "muon".

• Anatomy of a Frontier Model: Reading DeepSeek-V4 Through Part V

  Updated:16 Oct, 2026

  DeepSeek-V4 (April 2026) is two mixture-of-experts models with a million-token context: 1.6 trillion parameters with 49 billion active, and 284 billion with 13 billion active. Read through Part V, each unfamiliar part answers a cost the earlier posts measured: compressed and sparse attention for the KV cache, an indexer for the sparse selection, a widened residual stream kept stable by a doubly stochastic constraint, the Muon optimizer, FP4 experts, and on-policy distillation from more than ten teachers. The KV cache recomputed from the configuration: 2% of a standard model's at a million tokens.