Tag: ppo
All the articles with the tag "ppo".
Reinforcement Learning for Language Models: From Policy Gradient to PPO
Updated:When a score isn't differentiable, a model can still climb it with the log-derivative trick. Reinforcement learning's vocabulary mapped onto text, REINFORCE derived in three lines, why a baseline is free and essential, advantages and GAE, and PPO's clipped objective worked case by case. Then REINFORCE, REINFORCE with a baseline and PPO trained on a tiny autoregressive policy, with controls, ten seeds at two learning rates, and the memory bill for PPO on a language model: four models at once.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.