Tag: kl-divergence
All the articles with the tag "kl-divergence".
Knowledge Distillation: Learning from a Teacher's Distribution
Updated:A teacher's full next-token distribution carries far more information per example than the one token the text used. Soft labels and dark knowledge on a real teacher, temperature and the tau-squared factor derived and checked, forward vs reverse KL on a bimodal target, sequence-level and on-policy distillation, and distilled reasoning. Then Qwen2.5-0.5B trained on this blog from hard labels, from a 1.5B teacher's distribution, and from both, two seeds each, with what each learns and forgets.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.
VAEs and GANs: Two Ways to Sample What You Can't Write Down
Updated:A language model samples by writing the distribution down one token at a time. For images that means hundreds of thousands of sequential steps in an arbitrary pixel order, so two other answers grew up: learn an encoder and maximize a bound on the likelihood (the VAE), or learn a critic and fool it (the GAN). The ELBO derived and checked exactly on a model small enough to solve, the reparameterization trick, the KL term worked by hand, the GAN's optimal discriminator and its Jensen-Shannon value checked numerically, and all three models (VAE, GAN, WGAN-GP) trained on the same eight-mode test, where off-mode samples, a dropped mode and an outright training failure all show up, with a test of where the VAE's off-mode samples come from.