Tag: ablation

All the articles with the tag "ablation".

• Build a Tiny GPT, End to End, and Train It on This Blog

  Updated:20 Oct, 2026

  Every piece of the series so far, assembled into one 1.8-million-parameter GPT in 80 lines: embeddings, RoPE, RMSNorm, causal attention, a SwiGLU MLP, residual connections. Its parameters counted by formula and by the model, trained on this blog with the settings chosen on a dev slice, scored once on two held-out posts against the LSTM, Kneser–Ney and bz2. Then what each piece is worth, by taking it out, what the GPU is doing, what the model writes, and what it has memorized.