Tag: training
All the articles with the tag "training".
Knowledge Distillation: Learning from a Teacher's Distribution
Updated:A teacher's full next-token distribution carries far more information per example than the one token the text used. Soft labels and dark knowledge on a real teacher, temperature and the tau-squared factor derived and checked, forward vs reverse KL on a bimodal target, sequence-level and on-policy distillation, and distilled reasoning. Then Qwen2.5-0.5B trained on this blog from hard labels, from a 1.5B teacher's distribution, and from both, two seeds each, with what each learns and forgets.
Build a Tiny GPT, End to End, and Train It on This Blog
Updated:Every piece of the series so far, assembled into one 1.8-million-parameter GPT in 80 lines: embeddings, RoPE, RMSNorm, causal attention, a SwiGLU MLP, residual connections. Its parameters counted by formula and by the model, trained on this blog with the settings chosen on a dev slice, scored once on two held-out posts against the LSTM, Kneser–Ney and bz2. Then what each piece is worth, by taking it out, what the GPU is doing, what the model writes, and what it has memorized.
DPO: The Policy Is Its Own Reward Model
Updated:RLHF's KL-regularized objective has a closed-form optimal policy, and inverting it expresses the reward through the policy itself. Substituted into Bradley-Terry, the partition function cancels and preference learning becomes a logistic loss on log-probability ratios, with no reward model and no RL loop. The derivation step by step, its gradient, an exact check that in the right policy family DPO and RLHF are the same problem, a demonstration of DPO's collapse with a fully flexible policy and deterministic labels, IPO's bounded alternative, and KTO and ORPO.
GRPO and Reinforcement Learning from Verifiable Rewards
Updated:When an answer can be checked, the reward model goes away, and sampling a group of answers per question and comparing them within the group gives the advantage without a value network. Verifiable rewards, GRPO's group-relative advantage worked on four samples, its objective and KL estimator, the memory it saves, why groups that all agree teach nothing, and the length bias Dr. GRPO identified, measured on a toy where length never affects the reward.