Tag: training
All the articles with the tag "training".
Parameter-Efficient Fine-Tuning: LoRA, Prefix and Prompt Tuning
Updated:A low-rank update is often enough to fine-tune a model, so training only a low-rank part of it cuts trainable parameters and optimizer memory by orders of magnitude, and the result merges back into the weights for free. Intrinsic dimension, LoRA's shapes, initialization and scale, worked memory for an 8B model, QLoRA, prefix and prompt tuning and adapters, and serving many LoRAs at once. Then a hand-written LoRA on the fine-tune from Fine-Tuning, and What It Destroys at ranks 1, 8 and 64, two seeds each: what it learns and what it forgets next to full fine-tuning, how low-rank the full fine-tune's own update is, and a check that merging changes nothing.
Reinforcement Learning for Language Models: From Policy Gradient to PPO
Updated:When a score isn't differentiable, a model can still climb it with the log-derivative trick. Reinforcement learning's vocabulary mapped onto text, REINFORCE derived in three lines, why a baseline is free and essential, advantages and GAE, and PPO's clipped objective worked case by case. Then REINFORCE, REINFORCE with a baseline and PPO trained on a tiny autoregressive policy, with controls, ten seeds at two learning rates, and the memory bill for PPO on a language model: four models at once.
Reasoning Models and Test-Time Compute
Updated:A reasoning model spends generated tokens as compute: trained with verifiable rewards, it learns to think before answering, and gets better the longer it thinks. What test-time compute means, how RL produces long chains and distillation passes them to small models, thinking budgets, and what 1,000, 10,000 and 50,000 thinking tokens cost in KV cache and time on two real models. Then a 1.5B model on checkable word problems, two seeds: answering directly, reasoning step by step, and voting over up to 16 chains, with the tokens each one costs, then on harder eight-step problems, where voting stops helping and a verifier could still gain.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.