GRPO and Reinforcement Learning from Verifiable Rewards

RLHF’s reward model was needed because “helpful” can’t be computed. Some things can. A math answer can be compared with the known result, code can be run against tests, a response’s format can be parsed. Reinforcement learning from verifiable rewards (RLVR) trains on those checks directly: no human comparisons, no reward model, and no learned reward to overoptimize. It is how DeepSeek-R1 and other reasoning models (see Reasoning Models and Test-Time Compute) were trained, most often with GRPO, an algorithm that removes PPO’s value model.

RLVR with GRPO in one loop. A program checks each answer in place of a reward model, and the group’s mean and spread stand in for the value model, so only the policy and a frozen reference stay in memory.

At a glance

When an answer can be checked by a program, RL can train on the check directly: no human labels, no reward model. On rule-based rewards alone, DeepSeek-R1-Zero’s pass@1 on AIME 2024 rose from 15.6% to 71.0%.
GRPO uses a group of answers to the same question as its own baseline, so it needs no value model: two models in memory instead of PPO’s four.
A group whose answers all score the same teaches nothing: 66% of groups at $p = 0.9$ with 4 answers.
Per-answer length normalization biases length: on a toy where length never matters, it shortened right answers from 5.0 tokens to 2.3 to 2.8, and at update 50 wrong answers averaged 4.5 tokens with it, against 4.0 to 4.1 without.

Open Table of contents

Verifiable rewards
The group-relative advantage
What it saves
Groups that all agree teach nothing
An experiment: the length bias
What came after
Where this leads
Takeaways
References

Verifiable rewards

A verifiable reward is a program. For DeepSeek-R1 (DeepSeek-AI, 2025) it had two parts, both rule-based: an accuracy reward, checking the final answer of a math problem (in a required format, such as a box) or running code against test cases, and a format reward for putting the reasoning between thinking tags. They deliberately didn’t use a neural reward model, for two reasons: as RLHF showed, a large-scale RL run will find whatever a learned reward gets wrong, and retraining a reward model during the run would cost compute and complicate the pipeline.

A checker can be gamed too, just differently. A test suite that doesn’t cover a case rewards code that special-cases the tests; an answer checker that accepts any string containing the right number rewards listing many numbers. The reward is only as good as the check.

The group-relative advantage

Reinforcement Learning for Language Models’s PPO needs an advantage per token, and estimates it with a value model as large as the policy. GRPO (Shao et al., 2024) drops the value model. It was introduced in DeepSeekMath with a learned reward model, so it isn’t tied to verifiable rewards; DeepSeek-R1 later paired it with rule-based ones. For each question it samples a group of $G$ answers, scores each, and uses the group as its own baseline:

\hat{A}_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)},

the same advantage for every token of answer $i$ . Worked with $G = 4$ and rewards $(1, 0, 0, 1)$ : the mean is 0.5 and the standard deviation 0.5 (the population form, dividing by $G$ ), so the two right answers get $+1$ and the two wrong ones $-1$ . With rewards $(1, 0, 0, 0)$ , the mean is 0.25 and the deviation 0.433: the one right answer gets $+1.73$ and each wrong one $-0.58$ . A rare success is pushed up hard; a common failure gently. It is the mean baseline from Reinforcement Learning for Language Models, computed over answers to the same question, so that question difficulty cancels.

The worked examples. The group is its own baseline: a rare success is pushed up hard, a common failure gently.

The objective is PPO’s clipped one with this advantage, averaged over each answer’s tokens and over the group, plus a KL penalty to the reference policy added directly to the loss, rather than to the reward as in RLHF. The algorithm allows several passes over each batch ( $\mu$ in the paper’s Algorithm 1), which is what the ratio and clip are for. DeepSeekMath itself used one: “The policy model only has a single update following each exploration stage”, so the ratio is 1 when the gradient is taken and the clip never binds. For the penalty, GRPO estimates $\text{KL}(\pi_\theta \,\|\, \pi_\text{ref})$ per token from the sampled token’s probability ratio alone, the token being drawn from the policy (Schulman’s “k3” estimator):

\mathbb{D}_\text{KL} \approx \frac{\pi_\text{ref}(o_t)}{\pi_\theta(o_t)} - \log \frac{\pi_\text{ref}(o_t)}{\pi_\theta(o_t)} - 1 .

Its average under $\pi_\theta$ is exactly that KL, because the ratio $\pi_\text{ref} / \pi_\theta$ averages to 1 under $\pi_\theta$ , and it is never negative, because $x - 1 \ge \log x$ . Recent work (DAPO among it) drops the KL term altogether, on the grounds that a reasoning model is meant to move far from where it started.

What it saves

PPO for RLHF held four models (Reinforcement Learning for Language Models): policy, reference, reward model, value model. With a verifier and a group baseline, GRPO holds two: the policy, trained, and the reference, frozen. For a 7B policy in mixed-precision AdamW, 112 GB plus 14 GB, against about 250 GB for PPO with a 7B reward model. The cost moves elsewhere: every question now needs $G$ sampled answers per step (64 in DeepSeekMath), and for reasoning models those answers are thousands of tokens long, so generation dominates the run.

Groups that all agree teach nothing

If every answer in a group gets the same reward, every advantage is zero (and the standard deviation in the denominator is zero too, which implementations guard with a small constant). The group contributes no gradient. For a question the model gets right with probability $p$ , that happens with probability $p^G + (1 - p)^G$ . At $p = 0.9$ with $G = 4$ , that’s 66% of groups wasted; with $G = 16$ , 19%. Larger groups keep learning from questions the model mostly gets right or mostly gets wrong, which is where most questions end up as training goes on. DAPO (Yu et al., 2025) also filters such groups out and samples more questions to keep the batch full (“dynamic sampling”).

Groups with no signal. Larger groups keep learning from questions the model almost always gets right, or almost always gets wrong.

An experiment: the length bias

Two details of GRPO’s loss were questioned by Liu et al. (2025) in “Dr. GRPO”: dividing each answer’s loss by its own length, and dividing the advantage by the group’s standard deviation. The first, they argued, biases the model’s length: a short answer’s tokens each get a larger share of its advantage, so short right answers are reinforced more per token than long right ones, and long wrong answers are penalized less per token than short wrong ones. The second creates a difficulty bias: dividing by a small spread up-weights questions the model almost always gets right or almost always gets wrong. A toy where length can’t matter tests both, with two hybrids that change one detail at a time. To isolate the normalizations, every learner takes one gradient step per batch, so the ratio and clip never come into play, and there is no KL term. The first token can also be the end token, an empty answer, which is wrong. An answer is at most LMAX = 8 tokens, each from a vocabulary of V = 4 or the end token, and its length counts the end token. length_bias.py holds the policy as a table of logits per question, position and previous token, samples a group of answers for each of 16 questions, and rewards an answer whose first token is right. The four learners differ only in the advantage and the loss:

        R = (toks[:, 0] == answer[q]).float()
        Rg = R.view(Q, G)
        adv = (Rg - Rg.mean(1, keepdim=True))
        if method in ("GRPO", "GRPO, summed"):
            adv = adv / (Rg.std(1, keepdim=True, unbiased=False) + 1e-4)   # population std, as in the worked example
        adv = adv.reshape(-1)
        # log-probs of the sampled tokens, recomputed with gradients
        prev = torch.cat([torch.full((Q * G, 1), V + 1), toks[:, :-1]], 1)
        for t in range(1, LMAX):                                      # an end token doesn't change "previous"
            prev[:, t] = torch.where(toks[:, t - 1] == END, prev[:, t - 1], toks[:, t - 1])
        lp = logits[q[:, None], torch.arange(LMAX)[None], prev].log_softmax(-1).gather(-1, toks[..., None])[..., 0]
        mask = torch.arange(LMAX)[None] < length[:, None]
        tok_loss = -(adv[:, None] * lp) * mask
        if method in ("GRPO", "Dr. GRPO, per answer"):
            loss = (tok_loss.sum(1) / length).mean()
        else:
            loss = tok_loss.sum(1).mean() / LMAX
        opt.zero_grad(); loss.backward(); opt.step()

Full script: length_bias.py

group size 4: accuracy / mean length of right answers / of wrong answers, 5 seeds, at update
                                           1                    50                   100                   400
GRPO                      0.22   5.02   4.03    0.78   4.27   4.53    0.96   3.59   4.43    0.99   2.83   5.46
GRPO, summed              0.22   5.02   4.03    0.84   5.03   4.07    0.97   5.03   3.89    1.00   4.98   5.00
Dr. GRPO, per answer      0.22   5.02   4.03    0.78   4.28   4.50    0.96   3.58   4.44    0.99   2.83   5.15
Dr. GRPO                  0.22   5.02   4.03    0.84   5.02   4.03    0.97   5.04   3.83    1.00   5.00   5.46
group size 16: accuracy / mean length of right answers / of wrong answers, 5 seeds, at update
                                           1                    50                   100                   400
GRPO                      0.20   4.95   3.97    0.91   3.28   4.50    0.99   2.55   4.38    1.00   2.30   5.08
GRPO, summed              0.20   4.95   3.97    0.93   4.92   4.01    0.99   4.86   4.05    1.00   4.84   3.95
Dr. GRPO, per answer      0.20   4.95   3.97    0.91   3.26   4.51    0.98   2.62   4.57    1.00   2.35   4.58
Dr. GRPO                  0.20   4.95   3.97    0.92   4.93   3.97    0.98   4.88   3.99    1.00   4.85   3.40

The length bias, isolated. The reward never depends on length, yet per-answer normalization alone makes right answers shorter.

Data: the output above, means of 5 seeds

Every learner solves the task, from about 21% right to 99% or more. But the reward never depends on length, so an unbiased learner has no reason to change it, and they differ:

Right answers get shorter with per-answer normalization, and only with it. Under GRPO, from 5.0 tokens on average to 2.8 with groups of 4 and 2.3 with groups of 16. Dr. GRPO with per-answer normalization put back does the same (2.8 and 2.35); GRPO with summed losses doesn’t (5.0 and 4.8), and neither does Dr. GRPO (5.0 and 4.85). Dividing by the standard deviation changes nothing about length, as expected: it scales a whole group’s advantages equally. The shortening is the per-answer length normalization.
Wrong answers get longer with it. They start at 4.0 tokens, shorter than right ones because empty answers are wrong. At update 50, with accuracy near 80% for groups of 4, wrong answers average 4.5 tokens with per-answer normalization and stay at 4.0 to 4.1 without it, for both group sizes. By update 400 wrong answers are 1% or fewer, and their average length is too noisy to compare.
Larger groups learn faster per update, though they also sample four times as many answers: with 16 per question, accuracy is 98 to 99% by update 100, against 96 to 97% with 4, and the GRPO length drift is faster too (2.6 against 3.6 tokens at update 100).

In real reasoning training, the second half of the bias matters most: wrong answers that grow longer look like the model “thinking longer” while it isn’t getting anything right. DAPO divides by the total number of tokens in the batch; Dr. GRPO divides by a constant, the generation budget, as the code above does with LMAX. Neither divides each answer by its own length.

What came after

DAPO (Yu et al., 2025) collected four changes to GRPO for long reasoning: a higher upper clip bound than lower (“clip-higher”, $\epsilon$ 0.2 below and 0.28 above, so that unlikely but useful tokens can grow), dynamic sampling, a token-level loss, and a soft penalty on answers that run past the length limit. It also drops the KL penalty.
DeepSeek-R1-Zero (DeepSeek-AI, 2025) applied GRPO with only rule-based rewards to a base model, with no supervised fine-tuning first. Its pass@1 on AIME 2024 rose from 15.6% to 71.0%, and its answers grew longer as training went on, which DeepSeek read as the model learning to think longer. Liu et al. (2025) argue part of that growth is the length bias measured above: with Dr. GRPO, length grows far less at similar accuracy. Reasoning Models and Test-Time Compute picks up the story.

Where this leads

Reasoning Models and Test-Time Compute covers what RLVR trains a model to do with its tokens: spend more of them thinking before it answers, and how far that extra compute goes.

Takeaways

When an answer can be checked by a program, RL can train on the check directly: no human labels, no reward model. The check can still be gamed.
GRPO samples a group of answers per question and uses the group’s mean (and spread) as the baseline, so it needs no value model: two models in memory instead of PPO’s four.
A group in which every answer gets the same reward teaches nothing: $p^G + (1-p)^G$ of groups, 66% at $p = 0.9$ with 4 answers.
GRPO’s per-answer length normalization biases length. On a toy where length never affects the reward, it shortened right answers from 5.0 tokens to 2.3 to 2.8, and made wrong ones longer; without it, right answers stayed at about 5 tokens, with or without the standard deviation.

References

Start with the DeepSeekMath paper’s GRPO section, then Liu et al.’s Dr. GRPO for the biases measured here, and DAPO for the changes used in long reasoning runs.

Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024): GRPO, introduced with a learned reward model.
DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025): rule-based rewards, and R1-Zero trained by GRPO from a base model.
Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective (2025): Dr. GRPO, and the length and difficulty biases tested here.
Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025): DAPO’s four changes for long reasoning runs, including clip-higher and dynamic sampling.
Schulman, Approximating KL Divergence (2020): the k3 KL estimator GRPO uses.