DPO: The Policy Is Its Own Reward Model

RLHF trained a reward model on comparisons, then a policy against it with PPO: two training runs, four models in memory, and sampling in the loop. Rafailov et al. (2023) showed the middle step is unnecessary. The policy RLHF is looking for can be written in terms of the reward, so the reward can be written in terms of the policy, and the comparisons can train the policy directly. The title of their paper says it: your language model is secretly a reward model.

Table of contents

The derivation

Step 1: the optimal policy. RLHF derived the policy that maximizes reward minus $\beta$ times the KL divergence from the reference:

Step 2: invert it. Take logs and solve for the reward (assuming $\pi_\text{ref}$ gives every response some probability):

Any reward corresponds to an optimal policy, and this writes the reward through that policy; conversely, every policy is the optimal one for some reward. $Z(x)$ is still there, and it can’t be computed: it sums over every possible response.

Step 3: substitute into Bradley–Terry. The probability that $y_w$ is preferred to $y_l$ depends only on the difference of rewards, and $\beta \log Z(x)$ is the same for both responses to the same prompt, so it cancels:

Step 4: fit it. Replace $\pi^*$ with the policy being trained, $\pi_\theta$, and maximize the likelihood of the observed preferences, as RLHF did for the reward model. This is legitimate because the map runs both ways: any policy is the KL-optimal policy for the reward $\beta \log (\pi / \pi_\text{ref})$, up to a term depending only on $x$ that Bradley–Terry ignores, so searching over policies is the same as searching over rewards (Rafailov et al.’s Theorem 1). The DPO loss:

It is the reward model’s loss from RLHF, with the reward replaced by $\hat r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)}$, the implicit reward. Training the policy is training a reward model, one that comes with its own optimal policy attached. And it is Learning Is Optimization’s logistic regression once more: a sigmoid of a score difference, trained on binary outcomes.

Everything in it is computable. $\log \pi_\theta(y \mid x)$ is the sum of the response’s token log-probabilities, the same quantity as the SFT loss (Fine-Tuning, and What It Destroys); $\pi_\text{ref}$ is a frozen copy of the starting model, usually the SFT model, and its log-probabilities can be computed once, before training. The derivation assumes the compared responses were sampled from $\pi_\text{ref}$; when the preference data came from other models, Rafailov et al. first fine-tune on the preferred responses to shrink that mismatch.

The gradient, worked

Differentiating, with $\hat r$ the implicit reward and $\sigma'(z) = \sigma(z)(1 - \sigma(z))$:

Raise the log-probability of the preferred response, lower the other’s, weighted by the probability the implicit reward model currently gives the wrong ordering, $\sigma(\hat r_l - \hat r_w)$. Worked with $\beta = 0.1$: if the policy has already raised $y_w$‘s log-ratio by 2 and lowered $y_l$‘s by 1, the margin is $0.1 \times 3 = 0.3$, and the weight is $\sigma(-0.3) = 0.43$. A pair it ranks backwards by the same amount gets $\sigma(0.3) = 0.57$. A pair ranked correctly by an implicit-reward margin of 5 (a log-ratio margin of 50 at $\beta = 0.1$) gets $\sigma(-5) = 0.007$: nearly done.

What β does

$\beta$ is the same KL coefficient as in RLHF, but it shows up in two places. In the derivation it sets how far the optimal policy may move from the reference. In the loss, it scales the log-ratios inside the sigmoid: with small $\beta$, the policy must move its log-probabilities a long way before the loss is satisfied; with large $\beta$, a small move does it. $\beta$ is usually tuned per dataset. Unlike PPO, though, nothing in the DPO loss directly measures the KL divergence to the reference. The penalty acts only indirectly: the trained policy is the KL-regularized optimum for whatever reward DPO fits. If the data let that implicit reward become extreme, the “regularized” optimum can be arbitrarily far from the reference, and the loss says nothing about responses absent from the data. The experiment below shows when that happens.

An experiment: DPO, exactly

The script, dpo_toy.py, reuses the exact world of RLHF’s experiment: rlhf_toy.py is that post’s script, copied into this post’s folder, and dpo_toy.py takes its world, X, Y and sigmoid from it. It draws the 300 comparisons, then defines the policy, in two families, and the three losses:

def policy(params, kind):
    """log pi over all 512 responses. "features": log pi_ref + X theta, the
    family "RLHF"'s reward model lives in; "table": one free logit per
    response."""
    z = log_ref + (Xt @ params if kind == "features" else params)
    return z - torch.logsumexp(z, 0)

def loss_fn(params, kind, loss_kind, beta):
    lp = policy(params, kind)
    h = (lp[win] - log_ref[win]) - (lp[lose] - log_ref[lose])           # log-ratio margin
    if loss_kind == "DPO":
        return -F.logsigmoid(beta * h).mean()
    if loss_kind == "DPO, soft labels":                                 # labels = true preference probability
        return -(p_win * F.logsigmoid(beta * h) + (1 - p_win) * F.logsigmoid(-beta * h)).mean()
    return ((h - 1 / (2 * beta)) ** 2).mean()                           # IPO

Full script: dpo_toy.py

It then solves the feature family to convergence with L-BFGS, by both routes, and trains one free logit per response with each loss for 30,000 steps (Adam at learning rate 0.05, plain SGD at 1.0):

feature family, beta = 0.5: RLHF policy vs DPO policy, largest difference in any response's probability: 1e-08; KL from ref 2.04, gold gain 2.72

one free logit per response; the 300 comparisons mention 296 of 512 responses, 103 of which only ever won
loss             beta opt   steps     KL   gold  unseen  only won  top 2 chosen log ratio
DPO               0.5 Adam   1000   5.29   1.22    0.00      0.99   0.86     -10.3
DPO               0.5 Adam  30000   5.83   0.35    0.00      1.00   1.00    -138.4
DPO               0.5 SGD    1000   0.62   0.89    0.15      0.54   0.16       0.3
DPO               0.5 SGD   30000   3.54   1.51    0.00      0.95   0.51      -3.9
DPO, soft labels  0.5 Adam   1000   3.48   4.51    0.00      0.79   0.50      -3.3
DPO, soft labels  0.5 Adam  30000   3.76   4.75    0.00      0.91   0.50      -4.1
IPO               0.5 Adam   1000   0.41   0.42    0.18      0.43   0.08       0.1
IPO               0.5 Adam  30000   0.41   0.42    0.18      0.43   0.08       0.1
IPO               0.1 Adam   1000   4.77   0.65    0.00      0.98   0.88      -6.3
IPO               0.1 Adam  30000   4.69   0.66    0.00      0.98   0.91      -6.9

On RLHF’s exact world of 512 responses and 300 comparisons:

• In the right policy family, DPO and RLHF are the same computation. When the policy can only take the form $\log \pi_\text{ref} + X\theta$, the features RLHF’s reward model used, the DPO loss in $\theta$ is the reward model’s Bradley–Terry loss in $\beta\theta$. Solved to convergence, the two pipelines (fit an unregularized reward model, then compute the optimal policy; or run DPO) agree to $10^{-8}$ in every response’s probability. That isn’t an experimental finding, it is the derivation in concrete form: when the policy can express the optimum for every reward the reward model could learn, one loss replaces two training stages.
• With one free logit per response, DPO collapses. The 300 comparisons mention only 296 of the 512 responses, and 103 of them only ever win. With one comparison per pair the labels are deterministic, the data are perfectly separable, and the best margin is infinite: the loss approaches its floor only as every winner’s log-probability pulls infinitely far above its loser’s, which sends every loser’s probability to zero, and many such policies exist, so the KL regularization no longer picks one. That is the degenerate optimum Azar et al. (2024) described, and it is what happens: by step 1,000, 99% of the probability sits on responses that only ever won, and by step 30,000 all of it sits on two of them. The responses no comparison mentioned lose all theirs. The mean chosen log-ratio falls to $-138$ only because almost every chosen response lost its probability to those two. Which two win is an accident of the optimization, and the gold gain, 1.22 after 1,000 steps, falls to 0.35. The feature-family policy, which shares what it learns between responses with the same tokens, gained 2.72.
• The labels are the cause. The same table policy, trained on soft labels, the true preference probability for each pair, settles at a finite KL, 3.76 at step 30,000 with a gold gain of 4.75, close to the 3.48 and 4.51 of step 1,000. Real data never comes with the true probabilities, but it shows that the collapse comes from the 0/1 labels, not from flexibility as such. The exact numbers depend on the optimizer: Adam keeps moving parameters at a steady rate even as gradients vanish, and plain SGD drifts more slowly (KL 3.54, gold gain 1.51 at step 30,000), toward the same collapse.
• IPO bounds the margin, which isn’t the same as doing well. Azar et al.’s IPO loss replaces the log-sigmoid with a squared error that pulls the margin $h$ to a finite target, $(h - \frac{1}{2\beta})^2$. At $\beta = 0.5$ it settles, KL 0.41 and gold gain 0.42, unchanged from step 1,000 to 30,000, but with a target margin of only 1 it barely moves, and gains less than DPO stopped at step 1,000. At $\beta = 0.1$, a target of 5, the same table policy also piles onto a few winners: KL 4.7, 91% on two responses. IPO makes $\beta$ mean something again; it doesn’t make the data say more.

Real DPO runs show a related effect, the preferred answer’s probability falling even as the margin grows, called likelihood displacement (Razin et al., 2025). There the cause is shared parameters, which a table doesn’t have: when chosen and rejected answers have similar representations, pushing one down drags the other with it, and probability can leak to answers outside the data. A language model sits between the two extremes above: far from one free logit per response, but flexible enough that both failures show up in practice. The usual protections are early stopping on a held-out set, label smoothing (treating each label as, say, 90% sure, the soft-label idea), mixing in an SFT loss on the preferred answers (Pang et al., 2024), or a bounded loss like IPO’s.

Other failure modes, and other losses

• Length. Preferred answers in many datasets are longer, and the implicit reward learns length like any reward model would (RLHF’s overoptimization in a new place). Park et al. (2024) found DPO increases verbosity and proposed a length penalty in the loss.
• KTO (Ethayarajh et al., 2024) needs no pairs, only single answers marked good or bad, which are much easier to collect. Its loss borrows Kahneman and Tversky’s value function: the implicit reward is measured against a reference point (an estimate of the KL to the reference model), saturates through a sigmoid, and good and bad answers get separate weights (loss aversion). It still needs a reference model.
• ORPO (Hong et al., 2024) drops the reference model: it adds to the ordinary SFT loss a log-sigmoid term on the log odds ratio between the preferred and rejected answers (odds $= p / (1 - p)$, with $p$ the per-token average likelihood), so preference tuning and fine-tuning happen in one stage. SimPO (Meng et al., 2024) also drops the reference, using the length-normalized log-probability as the implicit reward.

What it costs, against PPO

DPO trains one model, the policy, against one frozen reference whose log-probabilities can be precomputed: for a 7B model, 112 GB of training state (Initialization and Optimizers), plus 14 GB for the reference if its log-probabilities are computed on the fly, against PPO’s two trained and two frozen models, about 250 GB (Reinforcement Learning for Language Models). And it never generates during training: the data is a fixed set of comparisons, which makes it a supervised fine-tune in cost and stability. The price is that it is offline: it only learns from responses someone already wrote, while PPO’s policy learns from its own samples as it changes. Head-to-head studies have found well-tuned PPO ahead of DPO, and on-policy data helping in general (Xu et al., 2024; Tajwar et al., 2024). DPO can also be run on-policy, sampling fresh pairs from the current policy each round and labeling them (online or iterative DPO). When the reward can be checked rather than learned, RL on the model’s own samples is the natural fit: GRPO and Reinforcement Learning from Verifiable Rewards’s setting.

Where this leads

• GRPO and Reinforcement Learning from Verifiable Rewards returns to RL, for rewards that can be checked: GRPO, which drops PPO’s value model.

Takeaways

• The KL-regularized RLHF optimum, inverted, writes the reward as $\beta \log \frac{\pi}{\pi_\text{ref}}$ plus a term that cancels in every comparison. Substituted into Bradley–Terry, it gives a supervised loss on the policy: DPO.
• DPO’s gradient raises the preferred answer and lowers the other, each in proportion to how wrong the implicit reward is about the pair.
• In a policy family that can express the optimum for every reward the reward model could learn, DPO and the RLHF pipeline are the same convex problem; on an exact toy they agreed to $10^{-8}$.
• With one free logit per response and one comparison per pair, DPO collapsed onto two responses that only ever won, and the gold reward gain fell from 1.22 to 0.35. Soft labels removed the collapse; IPO bounded it only by barely moving.
• DPO is cheaper and more stable than PPO, and offline: it learns only from responses it is given.

References

Start with the DPO paper’s section 4, which is the derivation here, then Azar et al. for why DPO can overfit and what IPO changes. Xu et al. compares DPO with well-tuned PPO.

• Rafailov, Sharma, Mitchell, Ermon, Manning and Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS, 2023): DPO, its derivation (section 4) and Theorem 1.
• Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences (AISTATS, 2024): IPO, and the degenerate optimum DPO reaches with deterministic labels.
• Ethayarajh, Xu, Muennighoff, Jurafsky and Kiela, KTO: Model Alignment as Prospect Theoretic Optimization (ICML, 2024): KTO, preference tuning from single answers marked good or bad.
• Hong, Lee and Thorne, ORPO: Monolithic Preference Optimization without Reference Model (EMNLP, 2024): ORPO, which drops the reference model.
• Park, Rafailov, Ermon and Finn, Disentangling Length from Quality in Direct Preference Optimization (Findings of ACL, 2024): DPO’s verbosity, and a length penalty in the loss.
• Xu et al., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML, 2024): DPO against well-tuned PPO.
• Tajwar et al., Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML, 2024): why on-policy data helps preference tuning.
• Meng, Xia and Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS, 2024): SimPO, a reference-free, length-normalized implicit reward.
• Pang et al., Iterative Reasoning Preference Optimization (NeurIPS, 2024): DPO with an added SFT loss.
• Razin, Malladi, Bhaskar, Chen, Arora and Hanin, Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization (ICLR, 2025): likelihood displacement in real DPO runs.