RLHF: From Christiano 2017 to InstructGPT to Today's Recipe

Reinforcement Learning for Language Models trained a policy on a reward it could compute. For “a helpful, honest answer” there is nothing to compute: nobody can write that function down. What people can do, quickly and fairly reliably, is look at two answers and say which is better. Reinforcement learning from human feedback (RLHF) turns those judgments into a reward, and trains the model on it with Reinforcement Learning for Language Models’s PPO.

Table of contents

From comparisons to a reward: Bradley–Terry

Bradley and Terry (1952) modeled paired comparisons with a hidden score per item: item $i$ beats item $j$ with probability

the sigmoid of a difference (Learning Is Optimization’s logistic function). Only differences matter: adding a constant to every score changes nothing. Fitting the scores is logistic regression on comparisons. Worked on three answers:

import numpy as np

# Bradley-Terry on three answers, A, B and C. People compared them in pairs:
# A beat B in 3 of 4 comparisons, B beat C in 3 of 4, A beat C in 4 of 5.
# Model: P(i beats j) = sigmoid(r_i - r_j). Fit r by maximizing the log-
# likelihood of the 13 outcomes. Only differences matter, so fix r_C = 0.
wins = {("A", "B"): (3, 1), ("B", "C"): (3, 1), ("A", "C"): (4, 1)}    # (i wins, j wins)
sigmoid = lambda z: 1 / (1 + np.exp(-z))
r = {"A": 0.0, "B": 0.0, "C": 0.0}
for _ in range(5000):
    grad = {k: 0.0 for k in r}
    for (i, j), (wi, wj) in wins.items():
        p = sigmoid(r[i] - r[j])
        grad[i] += wi * (1 - p) - wj * p                  # d/dr_i of wi log p + wj log(1 - p)
        grad[j] -= wi * (1 - p) - wj * p
    for k in ("A", "B"):
        r[k] += 0.05 * grad[k]
print("rewards: " + ", ".join(f"r_{k} = {v:+.2f}" for k, v in r.items()))
for (i, j), (wi, wj) in wins.items():
    print(f"P({i} beats {j}) = sigmoid({r[i]:.2f} - {r[j]:.2f}) = {sigmoid(r[i] - r[j]):.2f}   observed {wi}/{wi + wj} = {wi / (wi + wj):.2f}")

Full script: bradley_terry.py

rewards: r_A = +1.67, r_B = +0.84, r_C = +0.00
P(A beats B) = sigmoid(1.67 - 0.84) = 0.70   observed 3/4 = 0.75
P(B beats C) = sigmoid(0.84 - 0.00) = 0.70   observed 3/4 = 0.75
P(A beats C) = sigmoid(1.67 - 0.00) = 0.84   observed 4/5 = 0.80

The fit doesn’t reproduce every observed rate, and it shouldn’t. Three win rates are tied together by only two free differences, so the model finds the single set of scores most consistent with all of them: A beat C “only” 4 of 5 times, but A beat B and B beat C, so the model believes A is well ahead of C.

A reward model is the same thing with a neural network for the score: a language model whose next-token head is replaced by a linear head that outputs a single number, $r_\phi(x, y)$ for prompt $x$ and response $y$. Given a comparison where people preferred $y_w$ to $y_l$, its loss is the negative log-likelihood of that preference,

which is Learning Is Optimization’s binary cross-entropy with the reward difference as the logit. The reward model never sees an absolute score, so its scores mean something only relative to each other. InstructGPT kept all the pairs from one prompt’s ranking in the same batch element, each weighted by one over the number of pairs: shuffled, those correlated pairs made the reward model overfit within one epoch.

The 2017 paper: a backflip from comparisons

Christiano et al. (2017) showed this works for control. An agent in a simulator produced pairs of 1-to-2-second video clips of its behavior; a person picked the better clip; a reward model was fitted to those choices, and the agent was trained with RL on the reward model, with new comparisons collected as the agent improved. With 900 comparisons, under an hour of a person’s time, a simulated robot learned to do a backflip, a behavior nobody could easily specify as a reward function. With feedback on under 1% of the agent’s interactions, it nearly matched RL on the true reward on the robotics tasks, and came close on some Atari games but not others.

InstructGPT’s three steps

Ziegler et al. (2019) first applied the method to language models, adding the KL penalty below; Stiennon et al. (2020) scaled it to summarization, and Ouyang et al. (2022) to following instructions in general, with the recipe still used today:

1. Supervised fine-tuning (Fine-Tuning, and What It Destroys): about 13,000 prompts with answers written by labelers.
2. Reward model: for about 33,000 prompts, labelers ranked 4 to 9 of the SFT model’s answers; every pair from a ranking is one comparison. InstructGPT’s reward model had 6B parameters.
3. PPO against the reward model on about 31,000 prompts, with a penalty for moving away from the SFT model, below.

Labelers preferred the 1.3B InstructGPT model’s answers to those of the 175B GPT-3, a model over 100 times larger, even when GPT-3 was given a few-shot prompt. The usual reading is that RLHF mostly changes how a model uses what it already knows, rather than adding knowledge.

The KL leash, and the policy it implies

Maximize the reward model’s score with nothing else, and the policy finds whatever the reward model overrates: answers that are long, flattering, or odd in ways the reward model never saw in training. RLHF therefore maximizes reward minus a penalty for moving away from the reference policy $\pi_\text{ref}$ (the SFT model):

In practice the penalty is applied per token: each token’s reward gets $-\beta \log \frac{\pi(a_t \mid s_t)}{\pi_\text{ref}(a_t \mid s_t)}$, and the reward model’s score is added at the end. Summed over a sampled response, these terms are a one-sample estimate of the sequence’s KL divergence, so the two forms agree on average.

This objective has an exact solution, worth deriving because DPO is built on it. Write the objective for one prompt as a sum over responses, and divide by $\beta$:

where $\pi^*(y) = \pi_\text{ref}(y)\, e^{r(y)/\beta} / Z$ and $Z = \sum_y \pi_\text{ref}(y)\, e^{r(y)/\beta}$ makes it a distribution. $\log Z$ doesn’t depend on $\pi$, and a KL divergence is zero only when the two distributions are equal, so the best policy, over all possible policies (a real network can only approximate it), is

The reference policy, reweighted by exponentiated reward: a Boltzmann distribution with temperature $\beta$. Large $\beta$ keeps $\pi^*$ close to $\pi_\text{ref}$; small $\beta$ piles probability onto the highest reward. PPO can’t compute this distribution, because $Z$ sums over every possible response; it climbs toward it with sampled gradient steps on a parametric policy. On a space small enough to enumerate, it can be computed exactly, which the experiment below does.

An experiment: overoptimization, exactly

Everything in RLHF can be done exactly on a small enough space: 512 possible responses, a hidden gold reward standing in for what people actually prefer, 300 noisy comparisons drawn from it through Bradley–Terry, a reward model fitted to them, and the KL-regularized optimum $\pi^*$ for a range of $\beta$, with no sampling noise from PPO, in 100 random worlds of each kind. A response is 3 tokens from a vocabulary of 8, and X holds one one-hot feature per (position, token), so the reward model can’t represent an effect of two tokens together. The gold reward of the “random” world adds such pair effects, of either sign; the “in moderation” world also makes token 7 worth bonus once, nothing twice and -4 three times. The worlds, the reward model, and the loop over $\beta$:

def world(kind, seed, bonus=0.0):
    rng = np.random.default_rng(seed)
    w = rng.normal(0, 1, L * V)
    pair = rng.normal(0, 2.5, (V, V)) * (rng.random((V, V)) < 0.15)
    gold = X @ w + pair[Y[:, 0], Y[:, 2]]
    if kind == "in moderation":
        gold = gold + np.array([0.0, bonus, 0.0, -4.0])[(Y == 7).sum(1)]
    tok = np.exp(rng.normal(0, 0.5, (L, V)))
    tok /= tok.sum(1, keepdims=True)
    ref = tok[0, Y[:, 0]] * tok[1, Y[:, 1]] * tok[2, Y[:, 2]]
    return rng, gold, ref

def fit_reward_model(rng, gold, ref, n=300, steps=3000, lr=0.5, l2=1e-3):
    a, b = rng.choice(len(Y), n, p=ref), rng.choice(len(Y), n, p=ref)
    prefer_a = rng.random(n) < sigmoid(gold[a] - gold[b])          # noisy human labels
    win, lose = np.where(prefer_a, a, b), np.where(prefer_a, b, a)
    D = X[win] - X[lose]
    theta = np.zeros(L * V)
    for _ in range(steps):                                          # gradient ascent on the BT log-likelihood
        p = sigmoid(D @ theta)
        theta += lr * (D.T @ (1 - p) / n - l2 * theta)
    return X @ theta

betas = [3, 1, 0.5, 0.3, 0.2, 0.1, 0.05, 0.02]
for kind, bonus in (("random", 0.0), ("in moderation", 2.0), ("in moderation", 4.0)):
    rows = []                                                        # (seed, beta, KL, proxy gain, gold gain, 7s)
    for seed in range(100):
        rng, gold, ref = world(kind, seed, bonus)
        proxy = fit_reward_model(rng, gold, ref)
        for beta in betas:
            logits = np.log(ref) + proxy / beta
            logpi = logits - logits.max() - np.log(np.exp(logits - logits.max()).sum())
            pi = np.exp(logpi)
            rows.append((seed, beta, pi @ (logpi - np.log(ref)), pi @ proxy - ref @ proxy, pi @ gold - ref @ gold,
                         pi @ (Y == 7).sum(1)))

Full script: overoptimization.py

world "random": means over 100 worlds; the starting policy writes 0.37 sevens per answer
  beta  KL from ref  proxy gain  gold gain  7s per answer
     3         0.11        0.69       0.70           0.37
     1         0.88        1.87       1.88           0.37
   0.5         2.39        2.92       2.91           0.37
   0.3         3.76        3.46       3.45           0.36
   0.2         4.67        3.68       3.67           0.35
   0.1         5.64        3.83       3.82           0.34
  0.05         6.10        3.86       3.87           0.33
  0.02         6.37        3.87       3.88           0.32
worlds whose gold gain ends more than 0.5 below its peak: 6%

world "in moderation, +2 for one 7": means over 100 worlds; the starting policy writes 0.37 sevens per answer
  beta  KL from ref  proxy gain  gold gain  7s per answer
     3         0.11        0.68       0.74           0.47
     1         0.91        1.89       1.92           0.71
   0.5         2.42        2.95       2.75           0.95
   0.3         3.78        3.48       3.10           1.07
   0.2         4.66        3.70       3.24           1.11
   0.1         5.62        3.85       3.35           1.12
  0.05         6.10        3.88       3.40           1.12
  0.02         6.36        3.89       3.41           1.11
worlds whose gold gain ends more than 0.5 below its peak: 22%

world "in moderation, +4 for one 7": means over 100 worlds; the starting policy writes 0.37 sevens per answer
  beta  KL from ref  proxy gain  gold gain  7s per answer
     3         0.12        0.71       0.88           0.55
     1         1.06        2.15       1.92           1.02
   0.5         2.83        3.39       1.69           1.55
   0.3         4.20        3.93       1.05           1.85
   0.2         4.94        4.12       0.67           1.98
   0.1         5.66        4.22       0.35           2.07
  0.05         6.03        4.25       0.19           2.10
  0.02         6.29        4.26       0.10           2.11
worlds whose gold gain ends more than 0.5 below its peak: 71%

Read each table top to bottom as optimizing harder: $\beta$ falls, and the policy moves further from where it started (KL, in nats).

• When the reward model’s errors have no direction, optimizing it is mostly safe. In the “random” world the reward model can’t represent the pair effects, so it is wrong about many responses, but its errors point every which way. Gold reward rises with the proxy, from 0.70 to 3.88 on average, and only 6% of worlds end more than 0.5 below their best.
• A direction the policy can reach is what hurts. The “in moderation” worlds add one term: token 7 is good once and bad three times. The reward model, trained on the starting policy’s answers, which average about a third of a 7, learns “7 is good” at every position. When a single 7 is worth $+4$, that direction usually wins: the policy goes from 0.37 sevens per answer to 2.1, proxy reward climbs from 0.71 to 4.26, and gold reward peaks at 1.92, at $\beta = 1$ and about 1 nat of KL, then falls to 0.10. In 71% of worlds it ends more than 0.5 below its peak.
• Whether the policy goes there depends on the competition. When a single 7 is worth only $+2$, the policy writes about half as many of them: 1.1 sevens per answer on average, gold still rising on average (to 3.41), and 22% of worlds ending more than 0.5 below their peak. A likely reason, not measured here, is that the proxy’s weaker “7 is good” often loses to the best ordinary token at the same position, so the policy writes a 7 there only in some worlds.

The shape is qualitatively the one Gao et al. (2023) measured, in a setup much like this one: a large “gold” reward model labeled the comparisons that a smaller proxy reward model was trained on. Plotted against the square root of the KL divergence, proxy reward kept rising while gold reward rose, peaked and fell, with different curve shapes for best-of-$n$ and for RL. It is Goodhart’s law for RLHF, and the worlds above show what makes it bite: not that a reward model is wrong, but that it’s wrong in a direction the policy can push and that wins against the alternatives. In real RLHF the best-known such direction is length: reward models tend to prefer longer answers, and policies learn to pad (Singhal et al., 2024). Gao et al. also found that the KL penalty acted much like stopping early: it moved where the policy ended up on the curve, not the curve itself. Either one, chosen by watching a held-out measure of quality and not the reward model’s score, is what keeps the policy before the peak.

What it costs: the alignment tax

RLHF optimizes for what labelers prefer, which can cost abilities they weren’t judging. InstructGPT measured this as an alignment tax: performance on some public NLP benchmarks dropped after PPO. Mixing pretraining gradients back into the PPO updates (they called it PPO-ptx) recovered most of it, though not all, the same replay that limited forgetting in Fine-Tuning, and What It Destroys.

What changed since

• AI feedback. Constitutional AI (Bai et al., 2022) replaced human harmlessness labels entirely: a model critiqued and revised its own answers against a written list of principles (a supervised stage), then judged pairs of answers against those principles to train the preference model for RL, called RL from AI feedback (RLAIF). Human labels were still used for helpfulness.
• No reward model. DPO shows the reward model and the RL loop can be folded into one supervised loss on the comparisons.
• Verifiable rewards. For math and code, a checker replaces the reward model entirely, so there’s no learned reward to overoptimize (see GRPO and Reinforcement Learning from Verifiable Rewards), though verifiers can be gamed too.

Where this leads

• DPO derives DPO from the Boltzmann optimum above.
• GRPO and Reinforcement Learning from Verifiable Rewards replaces the reward model with a checker, and the value model with a group of samples.

Takeaways

• People can’t write down a reward for “helpful”, but they can compare. Bradley–Terry turns comparisons into scores, $P(i \succ j) = \sigma(r_i - r_j)$, and a reward model is that, with a network for $r$.
• RLHF is three steps, SFT, a reward model on comparisons, PPO against it. InstructGPT’s 1.3B model was preferred to the 175B GPT-3.
• The KL-regularized objective has an exact optimum: the reference policy reweighted by $e^{r/\beta}$. PPO approximates it; DPO uses it directly.
• A reward model’s error is mostly harmless when it has no direction and harmful when it has one the policy can reach: over 100 exact toy worlds, gold reward kept rising with random errors (6% of worlds fell back), and peaked and fell, 1.92 to 0.10 on average, when the reward model overrated a token that was good once and bad in excess (71% fell back).
• The KL penalty or early stopping, chosen against a held-out measure of quality, keeps the policy before that peak; replaying pretraining data pays back some of the alignment tax.

References

Start with the InstructGPT paper, then Gao et al. on overoptimization. Nathan Lambert’s RLHF Book covers the whole pipeline in depth.

• Bradley and Terry, Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons (Biometrika, 1952): the paired-comparison model a reward model is trained with.
• Christiano, Leike, Brown, Martic, Legg and Amodei, Deep Reinforcement Learning from Human Preferences (NeurIPS, 2017): a reward learned from clip comparisons, and the backflip from 900 of them.
• Ziegler et al., Fine-Tuning Language Models from Human Preferences (2019): the first RLHF on language models, with the KL penalty.
• Stiennon et al., Learning to summarize from human feedback (NeurIPS, 2020): the method scaled to summarization.
• Ouyang et al., Training language models to follow instructions with human feedback (NeurIPS, 2022): InstructGPT, its three steps, prompt counts and the alignment tax.
• Gao, Schulman and Hilton, Scaling Laws for Reward Model Overoptimization (ICML, 2023): overoptimization measured, proxy against gold reward as KL grows.
• Bai et al., Constitutional AI: Harmlessness from AI Feedback (arXiv, 2022): Constitutional AI and RL from AI feedback.
• Singhal, Goyal, Xu and Durrett, A Long Way to Go: Investigating Length Correlations in RLHF (COLM, 2024): length as a reward-hacking direction.