Tag: alignment
All the articles with the tag "alignment".
DPO: The Policy Is Its Own Reward Model
Updated:RLHF's KL-regularized objective has a closed-form optimal policy, and inverting it expresses the reward through the policy itself. Substituted into Bradley-Terry, the partition function cancels and preference learning becomes a logistic loss on log-probability ratios, with no reward model and no RL loop. The derivation step by step, its gradient, an exact check that in the right policy family DPO and RLHF are the same problem, a demonstration of DPO's collapse with a fully flexible policy and deterministic labels, IPO's bounded alternative, and KTO and ORPO.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.