Tag: preference-learning

All the articles with the tag "preference-learning".

• DPO: The Policy Is Its Own Reward Model

  Updated:19 Oct, 2026

  RLHF's KL-regularized objective has a closed-form optimal policy, and inverting it expresses the reward through the policy itself. Substituted into Bradley-Terry, the partition function cancels and preference learning becomes a logistic loss on log-probability ratios, with no reward model and no RL loop. The derivation step by step, its gradient, an exact check that in the right policy family DPO and RLHF are the same problem, a demonstration of DPO's collapse with a fully flexible policy and deterministic labels, IPO's bounded alternative, and KTO and ORPO.