Posts
All the articles I've posted.
Knowledge Distillation: Learning from a Teacher's Distribution
Updated:A teacher's full next-token distribution carries far more information per example than the one token the text used. Soft labels and dark knowledge on a real teacher, temperature and the tau-squared factor derived and checked, forward vs reverse KL on a bimodal target, sequence-level and on-policy distillation, and distilled reasoning. Then Qwen2.5-0.5B trained on this blog from hard labels, from a 1.5B teacher's distribution, and from both, two seeds each, with what each learns and forgets.
Speculative Decoding, Derived: Draft Models, N-grams, Medusa and EAGLE
Updated:Decoding is memory-bound, so checking k guessed tokens costs about as much as generating one. A rejection-sampling rule makes the output exactly the target model's distribution whatever the guesser, proved and checked by sampling. The expected tokens per step derived and checked, the speedup with the draft's cost, and an interactive simulator; draft models, prompt lookup, Medusa heads and EAGLE; and acceptance measured on real models for two kinds of task.
Build a Tiny GPT, End to End, and Train It on This Blog
Updated:Every piece of the series so far, assembled into one 1.8-million-parameter GPT in 80 lines: embeddings, RoPE, RMSNorm, causal attention, a SwiGLU MLP, residual connections. Its parameters counted by formula and by the model, trained on this blog with the settings chosen on a dev slice, scored once on two held-out posts against the LSTM, Kneser–Ney and bz2. Then what each piece is worth, by taking it out, what the GPU is doing, what the model writes, and what it has memorized.
Agent Architectures: ReAct, Plan-and-Execute and Reflection, Compared
Updated:Three ways to spend a model's tokens on thinking around tool use: interleave reasoning and actions (ReAct), plan everything first and then execute (plan-and-execute), or answer, critique and retry (reflection). How each works, the failure each is prone to, and all three run on the same 20 tool-use questions over a fictional database with a 3B model, measuring accuracy, tokens and time.