Tag: language-models
All the articles with the tag "language-models".
Build a Tiny GPT, End to End, and Train It on This Blog
Updated:Every piece of the series so far, assembled into one 1.8-million-parameter GPT in 80 lines: embeddings, RoPE, RMSNorm, causal attention, a SwiGLU MLP, residual connections. Its parameters counted by formula and by the model, trained on this blog with the settings chosen on a dev slice, scored once on two held-out posts against the LSTM, Kneser–Ney and bz2. Then what each piece is worth, by taking it out, what the GPU is doing, what the model writes, and what it has memorized.
What "Generative" Means: Autoregression, the Chain Rule, and Perplexity
Updated:A language model is one factorization, p(text) = the product of each token's probability given the ones before it, and training, sampling, caching and evaluation all follow from it. The chain rule worked on this blog, teacher forcing vs generation, what a small model writes when greedy and when sampling, perplexity derived from cross-entropy, why perplexity can't be compared across tokenizers and bits per byte can, measured with a character model and a BPE model on the same text.
Decoding: From a Distribution to Text
Updated:A language model outputs a probability for every possible next token; decoding is the policy that turns that into text. Greedy and beam search, temperature derived, top-k, top-p and min-p on a real model's distributions, the order the steps run in, repetition penalties, and why the same prompt can give different text. Each one measured with Qwen2.5-1.5B on twenty paragraphs of this blog, against what a person actually wrote next.