Tag: gqa

All the articles with the tag "gqa".

• How LLM Architecture Evolved, 2017–2026: A Short List of Substitutions

  Updated:19 Oct, 2026

  The Transformer of 2017 and a frontier model of 2026 share a skeleton: embeddings, a stack of attention and MLP blocks on a residual stream, an unembedding. What changed is a short list of substitutions, each made because it bought something measurable: pre-norm, RMSNorm, SwiGLU, RoPE, grouped and latent attention, mixtures of experts, and sparse or hybrid attention. Each one, with what it costs and saves, some measured on the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog. Then the scaling laws that decided how big to make them, Kaplan to Chinchilla to the replication that fixed it, and why models are now trained far past compute-optimal.