Tag: reward-model
All the articles with the tag "reward-model".
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.