How LLM Architecture Evolved, 2017–2026: A Short List of Substitutions

Part III: The Transformer built a Transformer the way it’s built in 2026, and along the way it kept saying “the 2017 paper did X; modern models do Y”. This post collects those changes in one place. The striking thing about them is how few there are. GPT-2’s diagram from 2019 still describes a 2026 model: a token embedding, $L$ blocks of attention and MLP reading from and adding to a residual stream (The Residual Stream), a final norm and an unembedding. What changed is a list of substitutions inside that skeleton, and each one stuck because it bought something that could be measured: stability, memory, speed, or quality at the same cost.

Table of contents

The substitutions

The rest of this post goes through them, grouped by what they’re for. Several have their own posts already, so here each gets its reason and its number.

Training stability: where the norms go, and which norms

Pre-norm moved the normalization off the residual stream’s main path (Dropout and Normalization and The Residual Stream). With post-norm, gradients near the output are large at initialization and training needs a careful warmup (Xiong et al., 2020); with pre-norm the identity path stays clean. Build a Tiny GPT, End to End, and Train It on This Blog measured the swap on its 4-layer model: post-norm cost 0.06–0.08 bits per byte, with that post’s caveats (a learning rate tuned for pre-norm, an unscaled embedding). The published evidence is that the gap grows with depth (Xiong et al., 2020), which is why almost every model since 2019 is pre-norm. Some recent ones add a second norm on each sublayer’s output before the add (Gemma 2 and later), or use only that one (OLMo 2); both keep the main path free of normalization.

RMSNorm (Zhang and Sennrich, 2019) drops LayerNorm’s mean subtraction and shift (see Dropout and Normalization). It matches LayerNorm’s quality and is simpler: one statistic instead of two, no shift parameters, a slightly cheaper backward pass. Normalization is memory-bound (How a GPU Runs Deep Learning), and in naive code RMSNorm saves a pass over memory; with the fused kernels production models use, the two cost about the same (Dropout and Normalization works this out). It won because it is no worse, no more expensive, and simpler. Dropping biases (T5, 2019; PaLM, 2022) removes parameters that bought little; PaLM reports it also improved training stability at scale. (Qwen2 and 2.5 are an exception: they keep biases on the query, key and value projections.)

The last group, QK-norm (normalizing queries and keys before their dot product; ViT-22B, Dehghani et al., 2023, and now OLMo 2, Qwen3 and Gemma 3), the z-loss (a small penalty on the log of the softmax’s normalizer, used by PaLM) and logit soft-capping (Gemma 2, which Gemma 3 replaced with QK-norm), all keep softmaxes from growing extreme in very large runs, where a single loss spike can cost days of compute. QK-norm is the one that has become standard. They matter at a scale this blog can’t reproduce.

Quality at the same cost: SwiGLU and RoPE

SwiGLU (Shazeer, 2020) replaces the MLP’s ReLU with a gated unit, shrinking the hidden width to $\tfrac83 D$ so the weights stay at $8D^2$ (see The Residual Stream). Shazeer’s matched-cost comparison found it better, and nearly every model since 2023 uses it or the GELU-gated variant. Newer models often go wider than $\tfrac83 D$: Llama 3 and Mistral use $3.5D$, Gemma 2 9B $4D$, Qwen2.5-7B about $5.3D$.

RoPE (Where Am I? Position in Attention, from Sinusoids to RoPE) moved position from an added input vector into a rotation of queries and keys, so attention scores depend on relative distance, with no parameters at all. (Some recent models leave a few layers without it, as in Llama 4.) Build a Tiny GPT, End to End, and Train It on This Blog measured it against a learned position table at the same context length: 0.1 bits per byte better at 4,000 steps, with the gap still closing.

Both can be measured on the tiny GPT of Build a Tiny GPT, End to End, and Train It on This Blog, trained the same way on the dev slice, two seeds each, swapping one piece at a time. GELU at $4D$ against SwiGLU at $\tfrac83 D$ has exactly the same parameter count; LayerNorm adds a bias per norm. train.py is the training script from Build a Tiny GPT, End to End, and Train It on This Blog, and this post’s gpt.py adds the mlp and norm options used here. mlp_norm_swaps.py runs train.py once per configuration:

# The tiny GPT's SwiGLU MLP and RMSNorm against a GELU MLP at 4D and against
# LayerNorm, one swap at a time, two seeds each, on the dev slice.
BASE = dict(kind="char", steps=4000, dropout=0.2, checkpoints=[1000, 2000, 3000, 4000])
for seed in (0, 1):
    train(f"SwiGLU + RMSNorm (baseline) s{seed}", **BASE, seed=seed)
    train(f"GELU MLP, 4D wide         s{seed}", **BASE, seed=seed, mlp="gelu")
    train(f"LayerNorm instead of RMS  s{seed}", **BASE, seed=seed, norm="layer")

Full script: mlp_norm_swaps.py

SwiGLU + RMSNorm (baseline) s0: 1000: 2.147 / 2000: 2.053 / 3000: 2.034 / 4000: 2.027  (61 passes)
GELU MLP, 4D wide         s0: 1000: 2.233 / 2000: 2.077 / 3000: 2.039 / 4000: 2.020  (61 passes)
LayerNorm instead of RMS  s0: 1000: 2.134 / 2000: 2.047 / 3000: 2.060 / 4000: 2.029  (61 passes)
SwiGLU + RMSNorm (baseline) s1: 1000: 2.140 / 2000: 2.030 / 3000: 2.023 / 4000: 2.006  (61 passes)
GELU MLP, 4D wide         s1: 1000: 2.246 / 2000: 2.074 / 3000: 2.022 / 4000: 2.009  (61 passes)
LayerNorm instead of RMS  s1: 1000: 2.127 / 2000: 2.051 / 3000: 2.028 / 4000: 2.017  (61 passes)

At this scale, with the 600 KB of text repeated 61 times, where training ends up is set by the data, and neither substitution changes it. After 4,000 steps GELU scores 2.020 and 2.009 against SwiGLU’s 2.027 and 2.006, and LayerNorm 2.029 and 2.017: differences of 0.01 or less, either way, the size of the gap between two seeds. What SwiGLU does change is the speed of getting there: after 1,000 steps (already about 15 passes over the data) it is ahead of GELU by 0.09 and 0.11 bits per byte on the two seeds, and by 0.02 to 0.04 after 2,000. In a run that sees each token about once, as large runs do, being ahead at a given step is the final result, and that is the regime Shazeer’s comparison measured. LayerNorm and RMSNorm tie within the seed spread at every checkpoint, as they should if RMSNorm’s case is its simplicity, not its quality. (All these runs use the constant learning rate, like the ablations in Build a Tiny GPT, End to End, and Train It on This Blog.)

These are small models on small data, and the published comparisons are at far larger scale. What the experiment shows is the kind of evidence each substitution rests on: a measurable gain at matched cost (SwiGLU, here in speed), the same quality more simply (RMSNorm), or stability (pre-norm, which Build a Tiny GPT, End to End, and Train It on This Blog measured).

Memory at inference: GQA and MLA

Every generated token reads the keys and values of every earlier token, the KV cache (The KV Cache), and for long contexts and large batches that cache, not the weights, fills GPU memory. Standard multi-head attention stores a key and a value per head per layer. Two substitutions cut that:

• Multi-query and grouped-query attention (Shazeer, 2019; Ainslie et al., 2023) let groups of query heads share one key head and one value head. With 8 groups for 64 query heads, the cache shrinks 8 times.
• Multi-head latent attention (MLA; DeepSeek-AI, 2024) caches one compressed vector per token and layer, from which every head’s keys and values are reconstructed (at inference, by folding the up-projections into the query and output projections rather than rebuilding them), plus a small key carrying RoPE (Where Am I? Position in Attention, from Sinusoids to RoPE explains why that part can’t be compressed).

inference_memory.py works out the cache per generated token for published configurations of each layout, and, for the mixture-of-experts models below, the parameters held against those used per token. The cache part:

# What the attention substitutions save: the KV cache per generated token
# ("The KV Cache"), in 16-bit numbers, for published configurations, and the
# attention weights per layer. MHA keeps a key and a value per head; GQA one
# per group of query heads; MLA a compressed vector per token plus a small
# rotated key (DeepSeek-AI, 2024).
def kv_bytes(layers, per_layer_elements, bytes_per=2):
    return layers * per_layer_elements * bytes_per

configs = [
    # name, layers, model width, query heads, key/value heads, head size
    ("GPT-3 175B (MHA)", 96, 12288, 96, 96, 128),
    ("Llama 2 70B (GQA, 8 groups)", 80, 8192, 64, 8, 128),
    ("Llama 3 8B (GQA, 8 groups)", 32, 4096, 32, 8, 128),
]
for name, L, D, nq, nkv, h in configs:
    cache = kv_bytes(L, 2 * nkv * h)
    attn = D * nq * h * 2 + D * nkv * h * 2                     # Q and output, then K and V
    print(f"{name:30s} KV cache {cache / 1024:7.0f} KiB per token; attention weights per layer "
          f"{attn / 1e6:6.1f}M (with full MHA: {4 * D * nq * h / 1e6:6.1f}M)")

# DeepSeek-V2: 60 layers, 128 heads of 128, KV compression 512, rotated key 64.
L, nh, dh, dc, dr = 60, 128, 128, 512, 64
mla, mha = kv_bytes(L, dc + dr), kv_bytes(L, 2 * nh * dh)
print(f"{'DeepSeek-V2 (MLA)':30s} KV cache {mla / 1024:7.0f} KiB per token; the same heads with MHA: "
      f"{mha / 1024:,.0f} KiB, {mha / mla:.0f} times as much")

Full script: inference_memory.py

GPT-3 175B (MHA)               KV cache    4608 KiB per token; attention weights per layer  604.0M (with full MHA:  604.0M)
Llama 2 70B (GQA, 8 groups)    KV cache     320 KiB per token; attention weights per layer  151.0M (with full MHA:  268.4M)
Llama 3 8B (GQA, 8 groups)     KV cache     128 KiB per token; attention weights per layer   41.9M (with full MHA:   67.1M)
DeepSeek-V2 (MLA)              KV cache      68 KiB per token; the same heads with MHA: 3,840 KiB, 57 times as much
Mixtral 8x7B                    47.0B parameters, 13.0B used per token (28%): per-token FLOPs of a 13B model, memory of a 47B one
DeepSeek-V2                    236.0B parameters, 21.0B used per token (9%): per-token FLOPs of a 21B model, memory of a 236B one

GPT-3’s full multi-head attention needs 4.5 MiB of cache per token, so 4,096 tokens of context for one user would be 18 GiB (GPT-3’s own context was 2,048). Llama 2 70B, a smaller but comparable width, needs 320 KiB per token with GQA. DeepSeek-V2’s MLA stores 68 KiB, 57 times less than the same heads with MHA would. GQA also removes attention weights (151M per layer instead of 268M for Llama 2 70B), but the point is the cache: its size decides how many users and how long a context fit on a GPU, which is the subject of The KV Cache and of Part XII: Inference engineering.

Parameters without FLOPs: mixtures of experts

A mixture of experts replaces each block’s MLP with many MLPs, the experts, and a router that sends each token to a few of them (see Mixture of Experts). The model holds all the experts’ parameters but uses only a few per token: Mixtral 8x7B has 47B parameters and uses 13B per token, DeepSeek-V2 236B and 21B. Per token it costs the FLOPs of the smaller number and the memory of the larger. At the largest scales, where quality is limited by parameters and compute is the budget, that trade has won: most frontier open models of 2025–2026 are MoE (DeepSeek-V3, Qwen3-235B, Llama 4, Kimi K2, gpt-oss, GLM-4.5).

Cost at long context: sliding-window, sparse and linear attention

Attention’s own work grows with the context, and The Residual Stream showed it overtaking the MLP’s at about $T = 8D$ tokens. The responses, covered in Attention Alternatives, keep full attention in some layers and replace it in others: sliding-window layers that see only a recent window (Mistral 7B used one in every layer; Gemma 2 alternates local and global layers, Gemma 3 uses five local layers of 1,024 tokens per global one, gpt-oss alternates with 128-token windows), sparse attention that selects which earlier tokens to read (DeepSeek-V3.2’s DSA), and linear-attention or state-space layers with a fixed-size state, interleaved with full attention (Jamba, Qwen3-Next, Kimi Linear, Nemotron-H). Not every hybrid stuck: MiniMax went back from a linear hybrid to full attention. The skeleton stays; only some layers’ mixing changes.

How big, on how much data: the scaling laws

The other half of a model’s design is its size, and the scaling laws are how that’s decided. Fit, across many training runs, a formula for the final loss of a model with $N$ parameters trained on $D$ tokens:

Here $D$ is tokens, not the model width it meant above. $E$ is the loss no model can beat (roughly, the entropy of the text under this tokenizer), and the two other terms shrink as the model and the data grow. Training compute is about $C = 6ND$ (Backpropagation, by Hand). For a fixed budget $C$, which $N$ minimizes the loss? Substituting $D = C / 6N$ and setting the derivative in $N$ to zero gives $\alpha A / N^{\alpha} = \beta B / D^{\beta}$, the two terms falling at matched rates, and so $N^* \propto C^{\beta / (\alpha + \beta)}$: the optimal model grows with the budget at a rate set by how fast each term falls.

The history is a correction, and then a correction of the correction.

• Kaplan et al. (2020) found the optimal model size growing fast with compute, $N^* \propto C^{0.73}$, and data only as $C^{0.27}$: spend extra compute mostly on a bigger model. GPT-3 (175B parameters on 300B tokens) followed that advice. (Later analyses add other reasons for Kaplan’s answer: it counted only non-embedding parameters, and its models were small; Pearce and Song, 2024.)
• Hoffmann et al. (2022), “Chinchilla”, found that parameters and data should grow about equally, $N^* \propto C^{0.5}$ by their first two methods (0.46 by the third), with a constant that works out to about 20 tokens per parameter. Their 70B model, trained on 1.4T tokens, beat their own 280B Gopher, trained on 300B, at the same compute. One reason for the difference, the Chinchilla paper argues, was the learning rate schedule: Kaplan used one fixed schedule for every run, whatever its number of tokens, while a cosine schedule matched to each run’s length gives a lower final loss (Build a Tiny GPT, End to End, and Train It on This Blog saw the same at small scale).
• Besiroglu et al. (2024) refit Chinchilla’s third method on data reconstructed from the paper’s own figure. The published constants fit that data badly, mostly because they were rounded in the paper (a data exponent of 0.28 for 0.2849) and partly because the original fit stopped before converging; the refit agrees with the paper’s other two methods.

The fit as printed, with its unrounded constants, and refit, worked through in chinchilla.py, which then applies them at three compute budgets and to Llama 3 8B:

# Compute-optimal training from a fitted scaling law,
#     L(N, D) = E + A / N^alpha + B / D^beta,
# the loss of a model with N parameters trained on D tokens (here D is
# tokens, not width), with training compute C = 6 N D ("Backpropagation, by Hand"). Three
# versions of one fit: Hoffmann et al.'s (2022, "Chinchilla", approach 3) as
# printed in the paper, the same with the unrounded constants from the
# paper's source, and Besiroglu et al.'s (2024) refit of the same data.
FITS = {"Hoffmann, as printed": dict(E=1.69, A=406.4, B=410.7, alpha=0.34, beta=0.28),
        "Hoffmann, unrounded": dict(E=1.6934, A=406.4, B=410.7, alpha=0.3392, beta=0.2849),
        "Besiroglu refit": dict(E=1.8172, A=482.01, B=2085.43, alpha=0.3478, beta=0.3658)}

def loss(f, N, D):
    return f["E"] + f["A"] / N ** f["alpha"] + f["B"] / D ** f["beta"]

def optimal(f, C):
    """Minimizing L subject to 6ND = C. Setting dL/dN = 0 along the
    constraint gives alpha A / N^alpha = beta B / D^beta, so
    N* = G (C/6)^(beta/(alpha+beta)), G = (alpha A / (beta B))^(1/(alpha+beta))."""
    a, b = f["alpha"], f["beta"]
    G = (a * f["A"] / (b * f["B"])) ** (1 / (a + b))
    N = G * (C / 6) ** (b / (a + b))
    return N, C / 6 / N

Full script: chinchilla.py

For Llama 3 8B, the script also finds, by bisection, the least training compute whose compute-optimal model reaches the 8B’s loss, and that model’s size: the “same loss” part of each line.

Hoffmann, as printed  tokens per parameter at C = 1e21 / 1e23 / 1e25 FLOPs:  50.1 /  78.2 / 122.1
Hoffmann, unrounded   tokens per parameter at C = 1e21 / 1e23 / 1e25 FLOPs:  34.0 /  50.7 /  75.7
Besiroglu refit       tokens per parameter at C = 1e21 / 1e23 / 1e25 FLOPs:  21.6 /  19.2 /  17.1

Llama 3 8B: 1,875 tokens per parameter, C = 7.2e+23 FLOPs
  Hoffmann, as printed  same compute: optimal  36B on  3.4T tokens, loss 1.923 vs the 8B's 1.949; same loss: a 26B model with 0.50x the 8B's training compute
  Hoffmann, unrounded   same compute: optimal  45B on  2.7T tokens, loss 1.911 vs the 8B's 1.944; same loss: a 29B model with 0.40x the 8B's training compute
  Besiroglu refit       same compute: optimal  81B on  1.5T tokens, loss 1.968 vs the 8B's 2.022; same loss: a 34B model with 0.18x the 8B's training compute
curve Hoffmann, as printed: 2:2.0266 4:1.9801 8:1.9485 16:1.9299 32:1.9228 64:1.9264 128:1.9405 256:1.9654
curve Hoffmann, unrounded: 2:2.0266 4:1.9777 8:1.9435 16:1.9221 32:1.9119 64:1.9121 128:1.9223 256:1.9426
curve Besiroglu refit: 2:2.1169 4:2.0622 8:2.0221 16:1.9941 32:1.9766 64:1.9688 128:1.9703 256:1.9813

Chinchilla’s published formula, taken literally, says 50 to 122 tokens per parameter. The same formula with its unrounded constants says 34 to 76: rounding a data exponent from 0.2849 to 0.28 moves the answer by a factor of 1.5. The refit gives 17 to 22, consistent with the paper’s other methods and its 20-tokens-per-parameter rule. Constants quoted to two digits are worth checking against the claim they’re quoted for.

Each curve is one fit’s predicted loss for models of different sizes, all trained with the same compute (smaller models on more tokens). The minimum is the compute-optimal size. Note how flat the valley is: the curves change by a few hundredths across a factor of eight in size.

And then everyone overtrained. Llama 3 8B was trained on about 15 trillion tokens, 1,875 per parameter, nearly a hundred times the compute-optimal ratio. Applied to it (with a caveat: these fits were made on DeepMind’s data and tokenizer, so the losses are illustrative, not Llama 3’s), every fit says a 36B to 81B model on fewer tokens would have reached a lower loss for the same compute. Turned around: a compute-optimal model of 26B to 34B would match the 8B’s loss with a fifth to a half of its training compute. Meta’s own fit, on its own data, makes the same point: the Llama 3 paper puts its 405B flagship near compute-optimal and says the smaller models were trained “much longer than is compute-optimal” on purpose, since they “perform better than compute-optimal models at the same inference budget”. Compute-optimal means optimal for training, and a popular model is trained once and used for billions of tokens. Each generated token costs about $2N$ FLOPs, so the 8B serves at roughly a quarter to a tenth of the cost of the compute-optimal model for the same budget. Paying extra once in training to save on every token served is the trade (Sardana et al., 2024, formalize it), and small open models since 2023 have been trained that way.

What hasn’t changed

Embeddings, causal attention with a softmax in at least some layers, an MLP per position, residual additions, a norm, next-token cross-entropy (with an extra multi-token prediction objective in some, like DeepSeek-V3), and warmup with decay: every 2026 language model still has them. Even the optimizer has mostly stayed AdamW, though Muon has started to replace it in some 2025 frontier runs (Kimi K2, GLM-4.5). Two smaller trends also run through these years: vocabularies grew, from 32K tokens (Llama 2) to 128K (Llama 3) and 256K (Gemma), and most large models stopped tying the embedding and unembedding (Gemma is an exception), while small ones usually still tie. The substitutions above changed what each piece costs, not what the pieces are.

Where this leads

Because the pieces stayed the same, each of the next posts can take one substitution and go deep:

• Mixture of Experts builds the router and the experts, and measures load balance.
• The KV Cache derives the cache that GQA and MLA shrink.
• Attention Alternatives covers the sliding-window, sparse and linear layers, and FlashAttention makes full attention itself cheaper in memory traffic.
• Anatomy of a Frontier Model reads one 2026 model that combines them all.

Takeaways

• The Transformer’s skeleton is unchanged since GPT-2. What changed is a short list of substitutions, each kept because it bought something measurable.
• Stability: pre-norm (0.06–0.08 bits per byte at only 4 layers, in Build a Tiny GPT, End to End, and Train It on This Blog), no biases, and at the largest scales QK-norm. Simplicity: RMSNorm, which matched LayerNorm here.
• Quality at the same cost: SwiGLU (it learned faster at the same cost here) and RoPE (it beat a learned table in Build a Tiny GPT, End to End, and Train It on This Blog).
• Inference memory: GQA and MLA shrink the KV cache by 4 to 8 times (the ratio of query heads to key-value heads) and 57 times against multi-head attention.
• Parameters without FLOPs: mixtures of experts use 9–28% of their parameters per token.
• Scaling laws set the size: compute-optimal is about 20 tokens per parameter (Chinchilla’s printed formula says more, mostly because its constants were rounded; the refit agrees). Models are now trained far past it, because inference, not training, dominates a popular model’s cost.

References

Start with Sebastian Raschka’s The Big LLM Architecture Comparison for diagrams of many 2024–2025 models side by side. Then read Hoffmann et al. and the Besiroglu refit together, and Sardana et al. for why models are trained past compute-optimal.

• Vaswani et al., Attention Is All You Need (NeurIPS, 2017): the 2017 Transformer, the left column of the table.
• Radford et al., Language Models are Unsupervised Multitask Learners (OpenAI, 2019): GPT-2, the skeleton a 2026 model still shares.
• Xiong et al., On Layer Normalization in the Transformer Architecture (ICML, 2020): pre-norm against post-norm, and why the gap grows with depth.
• Zhang and Sennrich, Root Mean Square Layer Normalization (NeurIPS, 2019): RMSNorm.
• Shazeer, GLU Variants Improve Transformer (2020): SwiGLU and its matched-cost comparison.
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021): RoPE.
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019): multi-query attention.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP, 2023): grouped-query attention.
• DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024): multi-head latent attention, and the configuration behind the 57 times.
• Jiang et al., Mixtral of Experts (2024): 47B parameters, 13B active.
• Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (JMLR, 2020): T5, an RMS-style norm and no biases.
• Chowdhery et al., PaLM: Scaling Language Modeling with Pathways (JMLR, 2023): SwiGLU, no biases, the z-loss, multi-query attention.
• Dehghani et al., Scaling Vision Transformers to 22 Billion Parameters (ICML, 2023): QK-norm for stability.
• Jiang et al., Mistral 7B (2023): sliding-window attention and GQA.
• Gemma Team, Gemma 2: Improving Open Language Models at a Practical Size (2024): local and global layers, logit soft-capping, sandwich norms.
• Lieber et al., Jamba: A Hybrid Transformer-Mamba Language Model (2024): interleaved attention and state-space layers.
• Pearce and Song, Reconciling Kaplan and Chinchilla Scaling Laws (2024): why Kaplan’s exponent came out larger.
• Kaplan et al., Scaling Laws for Neural Language Models (2020): the first scaling laws, $N^* \propto C^{0.73}$.
• Hoffmann et al., Training Compute-Optimal Large Language Models (NeurIPS, 2022): Chinchilla, the 20 tokens per parameter, and the fitted loss formula evaluated here.
• Besiroglu, Erdil, Barnett and You, Chinchilla Scaling: A replication attempt (2024): the refit of Chinchilla’s third method.
• Sardana, Portes, Doubov and Frankle, Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (ICML, 2024): why models are trained past compute-optimal.
• Grattafiori et al. (Meta AI), The Llama 3 Herd of Models (2024): 15T training tokens, GQA, base 500,000, and the quote on training past compute-optimal.