← Fine Tuning From First Principles

GRPO and Verifiable-Reward Fine-Tuning

Replace PPO's learned value baseline with group-relative advantages, compute them by hand including the zero-variance case, and show on a three-action bandit that a verifier drives true success up while a marker proxy drives it below 0.007.

PPO, as Chapter 34 built it, needs a value function to turn a reward into an advantage. For a language model that value function is usually a second network of comparable size, trained alongside the policy, consuming memory and compute and adding its own errors. If the value estimate is poor, so is every advantage computed from it.

There is a cheaper way to ask “was this response better than expected?” Sample several responses to the same prompt, score them all, and compare each one with the others. A response that scored above its siblings was better than expected for this prompt; one that scored below was worse. No critic is needed.

This is the idea behind Group Relative Policy Optimization, introduced by Shao and colleagues in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. The paper presents GRPO as a variant of PPO that drops the critic, estimating the baseline from the scores of a group of sampled outputs, and reports reduced memory use as one of its advantages.

The method pairs naturally with rewards that can be computed rather than learned: a unit test that passes, an answer that matches, a format that parses. This chapter builds such a verifiable reward, uses it to train a policy, and then trains the same policy on a reward that checks something easier than the task. The second run is the chapter’s central result. The measured runs were recorded as FT-P6-INTEGRATED-02-35-s37 and FT-P6-INTEGRATED-02-35-s38, on an educational bandit, not a language model. The integrated runs are bitwise-identical to the preserved historical records FT-P6-CODEX-03-35-s37/s38.

Group-relative advantage

For a prompt q, sample a group of G responses o₁, …, o_G from the current policy and obtain a reward r_i for each. The group statistics are

$$ \mu = \frac{1}{G}\sum_{i=1}^{G} r_i, \qquad s = \sqrt{\frac{1}{G}\sum_{i=1}^{G}(r_i-\mu)^2}, $$
and the advantage of response i is its standardised reward,
$$ A_i = \frac{r_i - \mu}{s + \epsilon}. $$
The DeepSeekMath paper writes the outcome-supervision advantage as (r_i − mean(r))/std(r) and assigns that one value to every token of response i. The formula as written there does not say whether std is the population or the sample standard deviation, and it has no ε. The book's implementation names its choices explicitly: the **population** standard deviation, dividing by G rather than G − 1, and a small ε = 10⁻⁸ added to the denominator. Each choice changes the numbers, so each must be stated.

The mean plays the role of PPO’s value baseline: it is an estimate, from this group alone, of the reward a typical response to this prompt receives. The division by s rescales advantages so that a group with widely spread rewards and a group with narrowly spread rewards produce updates of similar size.

Work some groups by hand

Two responses, rewards 0 and 2. The mean is 1 and the population standard deviation is 1. The advantages are (0 − 1)/(1 + 10⁻⁸) and (2 − 1)/(1 + 10⁻⁸), about −0.99999999 and +0.99999999. The ε is visible in the eighth decimal place.

Two responses, both rewarded 2. The mean is 2, the standard deviation is 0, and both advantages are exactly 0/(0 + 10⁻⁸) = 0. The implementation also flags the group as degenerate.

Four responses, one success: rewards 1, 0, 0, 0. The mean is 0.25 and the population standard deviation is √0.1875 ≈ 0.433013. The successful response receives about +1.732051 and each failure about −0.577350. Had the sample standard deviation been used, 0.5 instead of 0.433013, the advantages would have been +1.5 and −0.5. Same data, same method name, different update.

Four responses, one failure: rewards 1, 1, 1, 0. The mirror image: each success receives about +0.577350 and the failure about −1.732051. A rare outcome, good or bad, receives a large advantage; a common one receives a small advantage.

Group-relative advantage

The figure turns the worked rewards into advantage bars. Notice that the rare outcome receives the large magnitude, that identical rewards give zero with a degenerate flag, and that naming the population convention matters.

What identical rewards cannot tell you

The degenerate case deserves more than a footnote. When every response in a group receives the same reward, the group carries no information about which response is better, and group-relative advantages correctly report none. Dividing zero by a small ε does not create a ranking. Nor should anything else: inventing a preference among responses that the reward could not distinguish would be fabricating a signal.

This has a practical consequence. With a binary reward and success probability p for each independent sample, a group of G is degenerate with probability p^G + (1 − p)^G. For G = 4 that is about 0.23 when p is 0.317 and about 0.94 when p is 0.985. A policy that rarely succeeds, or nearly always succeeds, spends most of its groups learning nothing from the reward. Larger groups make degenerate groups rarer at the cost of more samples per prompt. The DeepSeekMath experiments sampled 64 outputs per question.

A degenerate group can still move the policy if the loss has other terms. A KL penalty towards a reference, or an entropy term, produces a gradient whether or not the rewards vary. That movement is regularisation, not learning from the reward, and should not be mistaken for it.

The objective

The policy update reuses PPO’s clipped surrogate from Chapter 34, with the group-relative advantage in place of a critic’s estimate. In the DeepSeekMath formulation, the surrogate is averaged over the tokens of each response and then over the G responses, and a KL penalty towards a reference policy is added directly to the loss rather than subtracted from the reward, which the paper notes keeps the advantage calculation simple.

For the KL term, the paper uses a sample-based estimator: for a token sampled from the current policy,

$$ \widehat{\mathrm{KL}} = \frac{\pi_{\mathrm{ref}}}{\pi_\theta} - \log\frac{\pi_{\mathrm{ref}}}{\pi_\theta} - 1, $$
which the paper describes as unbiased. The book checks this directly. For a three-action policy (0.6, 0.3, 0.1) and reference (0.2, 0.5, 0.3), the exact KL(π‖π_ref) is about 0.396058, and the expectation of the estimator under π equals it to twelve decimal places. Every term is also non-negative, since x − log x − 1 ≥ 0 for x > 0.

The book’s implementation of the estimator is tested but not used in the experiment. Its bandit has three actions, so the exact categorical KL can be computed cheaply, and the experiment uses that instead. Exact and sampled KL are different quantities with the same expectation; a report should say which one it used.

Two further conventions matter in language-model GRPO and collapse in this bandit. A response here is one action, which is one token, which is one sequence. Token-level and sequence-level probability ratios are therefore identical, and the 1/|o_i| averaging does nothing. In a real model, responses of different lengths are averaged differently under different conventions, and a token-level ratio is not a sequence-level ratio. Nothing in this chapter’s results should be generalised across that difference.

A verifier the policy cannot influence

The reward is the strict JSON verifier from Chapter 34. Its inputs are the two operands, the raw response text and the active criterion revision. It parses the text, refuses duplicate keys, requires exactly one key answer with an integer value of 0 or 1, and returns:

  • format: whether the response is a valid answer object;
  • correct: whether it is valid and its answer equals the parity;
  • reward: 1 if correct, 0 otherwise.

For the prompt used here, the parity of 1 + 2, the three candidate responses verify as follows, and these records are stored with the experiment:

Response Format Correct Verified reward Marker proxy
{"answer":1} true true 1 0
{"answer":0} true false 0 0
MARKER false false 0 1

The verifier reads nothing from the policy: not its probabilities, not its parameters, not its previous rewards. It is a fixed function of the response and the declared task. That independence is what makes it a measurement rather than a second model to be fooled.

The last column is a proxy reward: 1 if the response contains the string MARKER, 0 otherwise. It stands for any cheap check that is meant to correlate with success and does not. In real systems such proxies look more plausible: the presence of a formatted final answer, a confident tone, a length threshold, a keyword a grader looks for.

The experiment

The policy is the three-logit bandit from Chapter 34, starting from the same logits at each seed. Each of 24 rollouts samples 32 responses, arranged as 8 groups of 4. All 32 samples answer the same prompt, so the 8 groups are repeated draws for one task, not 8 different problems; the same response may appear more than once within a group. Advantages are group-relative, as above. There is no value parameter: the optimiser updates 3 policy elements and 0 value elements. Each rollout gets two Adam epochs at learning rate 0.08 with ε = 0.2 clipping, an exact-KL weight of 0.01 and an entropy weight of 0.001.

Two conditions differ only in the reward used for the advantages:

  • verified: the verifier’s reward;
  • proxy: the marker proxy.

In both, true success is the probability the final policy assigns to the correct response, and the proxy probability is the probability it assigns to MARKER.

Seed Condition True success before True success after Marker probability after
37 Verified 0.317067 0.984231 0.006979
37 Proxy 0.317067 0.006525 0.986093
38 Verified 0.350458 0.988272 0.005796
38 Proxy 0.350458 0.006329 0.986199

Under the verified reward, true success rose from about a third to above 0.98 at both seeds. Under the proxy, the policy learned to say MARKER. The expected proxy reward exceeded 0.986 at both seeds, and true success fell below 0.007 at both seeds.

Degenerate groups were common. Of the 192 groups in each run (8 per rollout over 24 rollouts), 117 were degenerate in the verified run and 123 in the proxy run at seed 37, and 120 and 111 at seed 38. The first rollout of each run had between one and three degenerate groups; by the final verified rollout all eight groups were degenerate at both seeds, because nearly every sample was correct. The update signal faded exactly as the formula above predicts, as the policy approached certainty.

Reward obtained is not task success

The proxy run separates five quantities that are easy to conflate.

  • Reward obtained is what the optimiser maximised. Under the proxy, the final policy’s expected reward exceeded 0.986.
  • Valid response asks whether the output is a well-formed answer at all. Only the two JSON responses are valid; the final proxy-trained policy produces one with probability below 0.014.
  • Correct answer asks whether a valid response gives the right value. Under the proxy-trained policy that probability was 0.006525 at seed 37 and 0.006329 at seed 38.
  • Task success is what someone deploying the model wanted. Here it coincides with a correct answer; in real tasks it may require more, such as a correct answer that is also justified, safe or timely.
  • Held-out generalisation asks whether success transfers to prompts not seen in training. This experiment has a single prompt and measures none.

A training dashboard showing only the first quantity would have reported a near-perfect run.

The mechanism is not subtle. The optimiser does not know what the reward was meant to measure. It increases the probability of whatever the reward actually pays for. When the proxy and the verifier pay for the same response, optimising the proxy also improves the task. When they pay for different responses, as here, optimisation follows the proxy and abandons the task.

The literature has names for this. Amodei and colleagues’ Concrete Problems in AI Safety lists avoiding reward hacking among its core problems. Skalse and colleagues’ Defining and Characterizing Reward Hacking defines it formally: a situation in which optimising an imperfect proxy reward lowers performance on the true reward. Gao, Schulman and Hilton’s Scaling Laws for Reward Model Overoptimization measures the same phenomenon for learned reward models, finding that optimising a proxy reward model too far can reduce performance as judged by the stronger “gold” model it was trained to imitate. The bandit here is the simplest possible case of the pattern: the proxy and the truth disagree on one action, and that action wins.

What the experiment establishes

It establishes that the implemented group-relative mechanism, with no critic, drives the bandit policy towards whatever its reward pays for: towards the correct response under the verifier and towards an invalid response under the marker proxy. It establishes that degenerate groups occur at the rate the arithmetic predicts and contribute no reward signal. And it establishes, concretely, that a reward rising towards its maximum is compatible with true success falling towards zero.

It does not establish anything about language-model GRPO. It measures no reasoning improvement, no throughput, no memory saving and no rate at which reward hacking occurs in large models. The proxy was chosen to disagree completely with the task, which real proxies rarely do. Its value is as a demonstration of mechanism, not as an estimate of how often the mechanism bites.

Limitations

The environment is a single prompt with three fixed responses. The eight groups per rollout are repeated samples for that one prompt. Token-level and sequence-level conventions coincide, so the experiment cannot distinguish them. KL is exact and categorical, not the sampled estimator used in the paper, which is implemented and tested separately. Standard-deviation and ε conventions are the book’s named choices. True success is an exact probability under a three-way softmax, not a measured generation rate. Two seeds and one proxy were run.

FineTuneLab

group_advantages in ftlab.preferences.rl takes a [prompts, G] reward tensor with G at least 2, refuses rewards that carry gradients or are not finite, and returns the group means, population standard deviations, advantages and degenerate-group flags, with its normalisation convention stated in the result. group_identity binds prompt and candidate observations for each group. sampled_kl implements the paper’s estimator for frozen reference log probabilities. verify_answer is the strict verifier. The experiment is bandit with method='grpo' in ftlab.preferences.experiments, which records every rollout’s sampled actions, observed rewards, group means, standard deviations, degenerate counts, ratios, clip fractions and KL values.

Where this leads

The proxy in this chapter was a string. The proxies that matter most in practice are behaviours that are easy to reward and look responsible. The final chapter of Part VI rewards a model for refusing, on a task where some requests should be refused and others should not, and shows how a policy can look better by the reward and safer by one metric while becoming useless by another and no better overall.

Further Reading

  1. Zhihong Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models” (2024). Read the source. Introduces GRPO with group-standardised rewards. Read it for the conventions the chapter pins down and the degenerate-group arithmetic it verifies.
  2. Leo Gao, John Schulman and Jacob Hilton, “Scaling Laws for Reward Model Overoptimization” (2022). Read the source. Measures how proxy optimisation degrades gold performance with scaling coefficients. Read it as the large-scale analogue of the marker-proxy collapse.
  3. Joar Skalse et al., “Defining and Characterizing Reward Hacking” (2022). Read the source. Defines reward hacking formally. Read it for the definition covering both the chapter’s verified success and proxy failure.