Reward Models, PPO and Classical RLHF
Separate the reward model, the policy and the value baseline; derive the pairwise reward loss and PPO's clipped surrogate for both advantage signs; and test both on tiny fixtures where the reward model fails exactly on the input pattern its training never showed.
The methods of Chapters 32 and 33 never write down a reward. They fold “which response is better” directly into the policy’s own probabilities. The older route, which made instruction-following language models practical, does the opposite. It turns comparisons into an explicit scoring function, and then trains a policy to pursue high scores.
Ouyang and colleagues’ Training language models to follow instructions with human feedback describes the pipeline that became standard: supervised fine-tuning on demonstrations written by labellers, a reward model trained on labellers’ rankings of the supervised model’s outputs, and reinforcement learning against that reward model. The approach builds on Christiano and colleagues’ Deep reinforcement learning from human preferences, which trained agents from human comparisons of short behaviour segments rather than from a hand-written reward function.
This chapter builds the two learned halves of that pipeline at toy scale. It fits a 49-parameter reward scorer to Chapter 31’s comparisons, and runs a PPO-style optimiser on a one-step decision problem with a verified reward. The reward scorer fails on held-out data in a way that is completely predictable once you look at what the training split contained. The optimiser succeeds, including under a stress control that was expected to expose failure but did not produce one (reported as observed in the PPO results).
The measured results were recorded as FT-P6-INTEGRATED-02-34-s37 and FT-P6-INTEGRATED-02-34-s38. The integrated runs are bitwise-identical to the preserved historical records FT-P6-CODEX-03-34-s37/s38. Neither experiment involves a language model, a human label or a full RLHF pipeline.
Three systems with three jobs
Classical RLHF trains up to three different functions, and confusion between them causes real errors.
- A reward model s(x, y) maps a prompt and a complete response to a scalar score. It is trained on preference data and then frozen. It answers “how good does this response look?”
- A policy π_θ(y | x) chooses responses. It is what is being fine-tuned and what is eventually deployed. It answers “what should I say?”
- A value function or baseline V(x) predicts the reward the policy will obtain from a state, on average. It is used only during training, to judge whether a particular outcome was better or worse than expected. It answers “how surprised should I be by this reward?”
The reward model and the value function both output a number, but they estimate different things. The reward model scores a specific response; the value function averages over the responses the current policy might produce, and changes as the policy changes. A reward model trained on yesterday’s comparisons does not become a value function by being evaluated on today’s policy.
flowchart LR
A[Demonstrations] --> B[SFT policy]
B --> C[Sampled outputs]
C --> D[Human comparisons]
D --> E[Reward model, then frozen]
E --> F[PPO against reward]
F --> G[Deployed policy]
F -.-> H[Value baseline, training only]
The pipeline separates the three jobs above. Notice that the reward model is frozen before PPO begins, and that the value baseline never scores a response for deployment.
The reward model’s loss
Under the Bradley–Terry model of Chapter 32, a score difference determines a preference probability: p(y_w ≻ y_l) = σ(s(y_w) − s(y_l)). Fitting the scorer by maximum likelihood on observed comparisons gives the pairwise ranking loss
Three properties matter in practice.
Only differences are identified. Adding the same constant to every score leaves every margin, and so the loss, unchanged; adding 100 to both scores in the example above changes the loss by exactly zero. A reward model’s absolute scores carry no meaning. A score of 5 is not “good” and a score of −1 is not “bad” unless someone fixes a reference point.
The scale is set by the data, not by a unit. Doubling every score doubles every margin and changes every predicted probability. The scale the scorer settles on is whatever made the training comparisons most likely, and a scorer trained on near-separable data is pushed towards ever larger margins.
Ties have their own loss. If a pair is recorded as a tie, the book’s implementation averages the loss of both directions, ½[log(1 + exp(−m)) + log(1 + exp(m))], which is minimised, at log 2, when the margin is zero. A margin of 1 costs about 0.813262 on a tie.
The reward model is trained to agree with recorded judgements. Every way Chapter 31 found for those judgements to be wrong, noisy or unrepresentative is inherited. And unlike a verifier, which checks a property of the response, a reward model predicts a judgement. On inputs unlike its training data, its prediction is an extrapolation.
A reward model on the comparison fixture
The engineering scorer is deliberately simple. Each candidate is represented by four features: the parity of a, the parity of b, the candidate’s value (0 or
- and a constant 1. A two-layer network, 4 inputs to 8 tanh units to 1 output, has 49 parameters. It is trained with Adam at learning rate 0.03 for 64 full-batch updates on the eight training comparisons, with the pairwise loss above. The scorer receives the parity bits directly; it does not read text, and it is not a language model.
| Seed | Train before → after | Validation before → after | Test before → after |
|---|---|---|---|
| 37 | 4/8 → 8/8 | 2/4 → 2/4 | 2/4 → 2/4 |
| 38 | 4/8 → 8/8 | 3/4 → 1/4 | 1/4 → 3/4 |
The training loss fell from 0.682 to 0.002 at seed 37 and from 0.699 to 0.009 at seed 38, measured at the first and last updates. The scorer ranks every training pair correctly at both seeds.
The held-out rows look erratic. At seed 37 nothing moved. At seed 38 test agreement improved from 1/4 to 3/4 while validation agreement fell from 3/4 to 1/4. A reader shown only the seed-38 test number would conclude that training helped; a reader shown only the validation number would conclude it hurt.
Both are explained by one fact recorded in Chapter 31. The criterion prefers the candidate equal to (a + b) mod 2, which, in terms of parity bits, is their exclusive-or. At each seed, one of the four parity combinations never appears in training: (0, 0) at seed 37 and (1, 1) at seed 38. After training, the scorer is correct on a held-out pair if and only if its parity combination appeared in training. That rule holds for all sixteen pairs at both seeds, across all three splits, without exception.
| Seed | Combination missing from training | Held-out pairs with it | Learned margin on those pairs |
|---|---|---|---|
| 37 | (0, 0) | 2 validation, 2 test | −0.864 |
| 38 | (1, 1) | 3 validation, 1 test | −3.567 |
The reason is the missing corner of exclusive-or. Three corners of a two-bit truth table do not determine the fourth. The data from (0, 1), (1, 0) and (1, 1) is equally consistent with exclusive-or and with other functions that disagree at (0, 0). The scorer fitted the three corners it saw and extrapolated to the fourth in the direction its architecture and initialisation happened to favour, which at both seeds was the wrong one. At seed 38 it was wrong with a margin of more than three and a half: confident, and mistaken.
The held-out counts then follow from arithmetic, not learning. At seed 37, half of each held-out split has the missing combination, so both splits score 2/4. At seed 38, three validation pairs and one test pair have it, so validation scores 1/4 and test 3/4. The “improvement” on the seed-38 test split and the “decline” on its validation split are the same model behaving the same way on differently composed four-pair sets.
The scorer was also evaluated on an input it was never designed for: a candidate value of −1 standing for a response that is not a valid answer at all. At seed 37 it scored that input 2.384, below the correct candidate at 5.841 but above the wrong one at −0.527. At seed 38 the corresponding scores were 0.927, 2.539 and −1.765. At both seeds the scorer ranked an invalid response above a valid wrong answer. Nothing in its training told it what an invalid response is worth, and it does not know that it does not know.
This is the distinction between a predicted reward and a verified one. A verifier for this task would check the parity and return the truth on every input, seen or unseen. The reward model returns an extrapolation from eight comparisons, and the policy optimising it cannot tell the difference.
From reward to policy gradient
Once a reward exists, the policy should make high-reward responses more likely. The policy-gradient identity says how. For a policy π_θ over responses y with reward R(y),
PPO’s clipped surrogate
Sampling is expensive, so one would like to take several gradient steps on each batch of samples. After the first step, though, the samples come from an older policy π_old, not the current one. Importance weighting corrects for this: the per-sample objective becomes ρA, with the probability ratio
The minimum behaves differently for the two signs of A, and the asymmetry is the point.
| Ratio ρ | A = +2 | A = −2 |
|---|---|---|
| 0.5 | 1.0, unclipped, gradient 2 | −1.6, clipped, gradient 0 |
| 0.9 | 1.8 | −1.8 |
| 1.0 | 2.0 | −2.0 |
| 1.1 | 2.2 | −2.2 |
| 1.5 | 2.4, clipped, gradient 0 | −3.0, unclipped, gradient −2 |
Take ρ = 1.5: the new policy assigns probability 0.3 to an action that the old policy gave 0.2. With A = +2 the unclipped term is 3 and the clipped term 2.4. The minimum chooses 2.4, which does not depend on θ, so this sample contributes no gradient. The policy has already raised this good action’s probability by more than 20%, and PPO stops it raising it further on this batch. With A = −2 the unclipped term is −3 and the clipped term −2.4. The minimum chooses −3, the unclipped term, so the gradient is still active. The policy has raised the probability of a bad action, and PPO lets the gradient pull it back down without limit.
The mirror case is ρ = 0.5. A good action whose probability has fallen is unclipped and still pushed up. A bad action whose probability has already fallen by more than 20% is clipped and left alone. In every case, clipping removes the incentive to move further in the direction that already improved the objective, and never removes the incentive to undo a step that made it worse.
Value, entropy and the reference penalty
Full PPO training combines several terms. A value loss fits the baseline to observed returns,
For multi-step problems, advantages are usually computed with Generalised Advantage Estimation, from Schulman and colleagues’ High-Dimensional Continuous Control Using Generalized Advantage Estimation. With temporal-difference residual δ_t = r_t + γV_(t+1) − V_t, the advantage is the discounted sum A_t = δ_t + γλA_(t+1), where λ trades bias for variance. For a three-step trajectory with rewards (0, 0, 1), values (0.5, 0.6, 0.7), γ = 0.99 and λ = 0.95, ending in a terminal state, the advantages are about 0.446829, 0.375150 and 0.300000. The book implements this recurrence for one complete trajectory and tests it, but the experiment below has a single step, so GAE does not enter it. A one-step problem has nothing to assign credit across.
The objective used in the experiment is
A PPO-style bandit with a verified reward
The decision problem is a bandit: one fixed prompt, asking for the parity of 1 + 2, and three possible responses:
{"answer":1}, a valid JSON object with the correct answer;{"answer":0}, a valid JSON object with the wrong answer;MARKER, which is not JSON at all.
The reward comes from a strict JSON verifier. It parses the response, refuses
duplicate keys, requires exactly one key answer with an integer value of 0
or 1, and returns 1 only if the value equals the parity under the active
policy revision. The three responses receive rewards 1, 0 and 0.
The policy is three logits, one per response. The baseline is a single trainable scalar. Each of 24 rollouts samples 32 responses from the current policy, computes advantages against the baseline, and takes two Adam steps at learning rate 0.08 on the loss above. True success is the probability the final policy assigns to the correct response, computed exactly from its softmax rather than counted from samples.
Six conditions share each seed’s starting logits:
- verified: the rewards and trained baseline described above;
- no baseline: the baseline is fixed at zero;
- wrong baseline: the baseline is fixed at 5, larger than any reward;
- noisy: every third sampled reward is flipped before use;
- excessive: the learning rate is raised from 0.08 to 1.0;
- learned: the rewards are the seed’s reward-model scores for the correct answer, the wrong answer and the invalid response.
| Condition | True success, seed 37 | True success, seed 38 |
|---|---|---|
| Before training | 0.317067 | 0.350458 |
| Verified | 0.985388 | 0.988072 |
| No baseline | 0.984151 | 0.987945 |
| Wrong baseline | 0.848974 | 0.953593 |
| Noisy reward | 0.976475 | 0.969997 |
| Excessive learning rate | 0.999995 | 1.000000 |
| Learned reward | 0.981388 | 0.978598 |
Every condition improved true success substantially. Three observations need more than the table.
The clip never engaged in the ordinary runs. In every update of every condition except the excessive one, the fraction of sampled ratios outside [0.8, 1.2] was zero. At this learning rate, two epochs did not move any action’s probability by 20% relative to the rollout policy. The clipped surrogate was therefore identical to the unclipped one, and these runs tested a baseline-corrected, KL-regularised policy gradient. The asymmetry worked out above was not exercised by them.
The excessive condition did not fail. At learning rate 1.0 the clip did engage: in at least one update every sampled ratio fell outside the interval. The policy nevertheless converged to the correct response, with true success 0.999995 at seed 37 and above 0.9999997 at seed 38. Large steps are dangerous when they carry the policy away from where the reward is trustworthy, or when the reward is wrong. Here the reward is exactly right and the problem has three actions, so a large step in the right direction is simply a fast route to the answer. The control was designed to expose a failure and did not produce one, and it is reported as it happened.
The wrong baseline still learned, more slowly. With a baseline of 5, every advantage is negative: −4 for the correct response and −5 for the others. Every sampled response is pushed down, but the wrong ones are pushed down harder, and under a softmax, lowering two logits raises the probability of the third. The expected gradient is unchanged by a constant baseline; its variance is not, and with finite samples the policy reached 0.848974 and 0.953593 rather than the values above 0.98 reached with the trained baseline. The zero baseline did about as well as the trained one, because with rewards of 0 and 1 a zero baseline still gives positive advantages only to correct responses.
The learned condition is worth reading alongside the reward model’s failures. The scores it used ranked the invalid response above the wrong answer at both seeds. That error did no harm here, because the scores also ranked the correct answer highest, and a policy optimising them converges to whichever action scores highest. Had the scorer’s extrapolation put the invalid response at the top, the same optimiser would have pursued it just as efficiently. The next chapter runs exactly that experiment, with a hand-written proxy in place of the learned one.
What the experiments establish
The reward-model experiment establishes, at toy scale, that a scorer can fit its training comparisons perfectly and fail on held-out comparisons in a way fully determined by what the training split omitted. It also shows that four held-out pairs can make the same behaviour look like improvement on one split and decline on another. The scorer’s scores are uncalibrated: their offset is unidentified, their scale is whatever fitting produced, and its output for an invalid response is meaningless.
The bandit experiment establishes that the implemented PPO arithmetic drives a three-action policy towards the response a verifier rewards, under every control tried, and that clipping engaged only when the learning rate was raised by a factor of 12.5. It does not establish that PPO is stable for language models, that a baseline matters little in general, or that large learning rates are safe.
Limitations
The reward scorer reads four hand-built features, has 49 parameters and was trained on eight comparisons; it is not a language-model reward model and its held-out results reflect the split’s composition. The policy experiment is a one-step bandit with three actions, one prompt and an exact reward; it has no language rollouts, no token-level credit assignment, no human labels, no distributed sampling and no long horizon. The clip constraint was inactive in all ordinary runs. GAE was implemented and tested on a separate trajectory, not used. A hash binding of each rollout’s old policy and sampled actions is a consistency check within the run, not authentication of the execution.
FineTuneLab
Reward-model and policy arithmetic live in ftlab.preferences.
objectives.reward_loss implements the pairwise ranking loss with optional tie
handling. In rl, surrogate computes the clipped objective from detached old
log probabilities and advantages, refuses unbounded ratios and reports the clip
fraction; value_loss, entropy and categorical_kl provide the other
terms; gae implements the recurrence for one complete trajectory and refuses
to bootstrap past a terminal state; verify_answer is the strict JSON
verifier; and rollout_binding refuses a second epoch whose rollout identity
differs from the frozen one. The experiments are reward_experiment and
bandit with method='ppo' in ftlab.preferences.experiments.
Where this leads
PPO needs a value function to turn rewards into advantages, and in language modelling that value function is a second network as large as the policy. The next chapter replaces it with something cheaper: sample several responses to the same prompt and judge each one against the others. It also replaces the learned reward with a verifier, and then deliberately replaces the verifier with a proxy that is easier to satisfy than the task.
Further Reading
- Long Ouyang et al., “Training language models to follow instructions with human feedback” (2022). Read the source. Builds InstructGPT via demonstration tuning, reward ranking and PPO. Read it as the two-stage architecture the chapter rebuilds at toy scale; also recommended in Chapter 16 for the SFT separation.
- John Schulman et al., “Proximal Policy Optimization Algorithms” (2017). Read the source. Introduces the clipped surrogate enabling multiple epochs per batch. Read it for the asymmetric clipping the chapter works through by hand.
- John Schulman et al., “High-Dimensional Continuous Control Using Generalized Advantage Estimation” (2015). Read the source. Estimates advantages via exponentially weighted residuals. Read it for the multi-step machinery the chapter tests but excludes from its bandit.