Tag: architecture
All the articles with the tag "architecture".
How LLM Architecture Evolved, 2017–2026: A Short List of Substitutions
Updated:The Transformer of 2017 and a frontier model of 2026 share a skeleton: embeddings, a stack of attention and MLP blocks on a residual stream, an unembedding. What changed is a short list of substitutions, each made because it bought something measurable: pre-norm, RMSNorm, SwiGLU, RoPE, grouped and latent attention, mixtures of experts, and sparse or hybrid attention. Each one, with what it costs and saves, some measured on the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog. Then the scaling laws that decided how big to make them, Kaplan to Chinchilla to the replication that fixed it, and why models are now trained far past compute-optimal.
Anatomy of a Frontier Model: Reading DeepSeek-V4 Through Part V
Updated:DeepSeek-V4 (April 2026) is two mixture-of-experts models with a million-token context: 1.6 trillion parameters with 49 billion active, and 284 billion with 13 billion active. Read through Part V, each unfamiliar part answers a cost the earlier posts measured: compressed and sparse attention for the KV cache, an indexer for the sparse selection, a widened residual stream kept stable by a doubly stochastic constraint, the Muon optimizer, FP4 experts, and on-policy distillation from more than ten teachers. The KV cache recomputed from the configuration: 2% of a standard model's at a million tokens.
Mixture of Experts: More Parameters, Same FLOPs
Updated:A mixture of experts replaces a Transformer's MLP with many MLPs and a router that sends each token to a few of them, so a model can hold far more parameters than it uses per token. The router worked through, the load-balancing loss derived, DeepSeek-V3's bias balancing, capacity and dropped tokens, and shared and fine-grained experts. Then an MoE version of the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog, trained four ways, the parameter counts of Mixtral and DeepSeek-V3 recomputed from their configs, and what experts cost at serving time: memory, all-to-all traffic, and batches big enough to keep experts busy.