Preference Rankers — Learning Which Answer Is Better
Preference Rankers — Learning Which Answer Is Better
So far in Models From First Principles, we have mostly trained models by telling them what the answer should be.
MR.Q took a context and a response and produced a number:
context + response
↓
model
↓
0.82
That number might mean quality.
Or usefulness.
Or reward.
But there is an awkward question hiding underneath that architecture:
Where did the 0.82 come from?
Someone has to provide the target.
And for many of the things we actually care about, that is surprisingly difficult.
Suppose I give you this sentence:
The system performs a comprehensive analysis of the available information.
Then I ask:
How good is this sentence on a scale from 0 to 1?
Is it 0.43?
0.61?
0.72?
There is no obvious answer.
Now suppose I give you two sentences:
A
The system analyzes the available information.
B
The system performs a comprehensive analysis of the available information.
and ask:
Which would you rather keep?
That is much easier.
This gives us a different kind of model.
Instead of learning:
candidate
↓
score
we learn:
candidate A
candidate B
↓
which is better?
That is a preference ranker.
And it turns out to be extraordinarily useful.
Relative Judgments Are Often Easier Than Absolute Ones
Imagine that we are training a model to understand writing quality.
One approach is to construct a dataset like this:
sentence quality
----------------------------------------------------
"The dog ran." 0.71
"The canine proceeded rapidly." 0.42
"The exhausted dog sprinted toward the gate." 0.84
But these numbers are artificial.
Why exactly is the third sentence 0.84 rather than 0.79?
The labels imply a precision that our judgment probably does not contain.
Instead, we can collect:
A B
-----------------------------------------------------------------
"The dog ran." "The canine proceeded rapidly."
preferred = A
Then another pair:
A B
-----------------------------------------------------------------
"The exhausted dog sprinted..." "The dog ran."
preferred = A
We no longer need to know the absolute quality of any sentence.
We only need to know:
A > B
The model can learn from the ordering.
This is the central idea behind preference learning.
The Model We Use in Writer
Writer uses a small pairwise neural ranker.
The core model is surprisingly simple:
class PreferenceRanker(nn.Module):
def __init__(self, embedding_dim=768, hidden_dim=256):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(embedding_dim, hidden_dim),
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(hidden_dim, hidden_dim),
)
self.comparator = nn.Sequential(
nn.Linear(hidden_dim * 2, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, emb_a, emb_b):
feat_a = self.encoder(emb_a)
feat_b = self.encoder(emb_b)
combined = torch.cat([feat_a, feat_b], dim=1)
return self.comparator(combined).squeeze(1)
There are really only two components:
shared encoder
comparator
The architecture looks like this:
candidate A ──→ shared encoder ──→ feature A ──┐
│
├──→ comparator ──→ preference logit
│
candidate B ──→ shared encoder ──→ feature B ──┘
The important word is shared.
Both candidates pass through the same encoder.
We do not have:
encoder_A
encoder_B
We have:
encoder
encoder
with the same parameters.
That matters.
Why Share the Encoder?
Suppose candidate A had one neural network and candidate B had another.
The model could accidentally learn:
things arriving on the left mean one thing
things arriving on the right mean another
But A and B are not fundamentally different kinds of objects.
They are two candidates playing temporary roles.
Today:
A = original
B = revision
Tomorrow:
A = revision
B = another revision
We therefore want both candidates represented in the same learned feature space.
If the encoder is:
[ E(x) ]
then we calculate:
[ h_A = E(A) ]
and:
[ h_B = E(B) ]
using exactly the same function (E).
Then the comparator receives:
[ [h_A \Vert h_B] ]
where (\Vert) means concatenation.
The comparator learns a function:
[ C(h_A, h_B) ]
and produces one logit:
[ z = C(h_A, h_B) ]
That logit answers:
How strongly should A be preferred to B?
This Is a Siamese Network
Architectures like this are often called Siamese networks.
The name sounds more exotic than the mechanism.
It simply means that the two inputs travel through identical networks that share their parameters.
┌─────────────┐
A ────────→│ │────→ h_A
│ encoder │
B ────────→│ │────→ h_B
└─────────────┘
same weights
The interesting part is not that we have two encoders.
We really have one encoder used twice.
This creates comparable representations.
But Writer Does Not Rank the Sentence Alone
There is another important detail.
Whether one answer is better than another often depends on the goal.
Consider:
A
He left.
B
Without warning, he abandoned the silent room and disappeared into the darkness beyond the door.
Which is better?
There is no answer.
If the goal is:
Make this concise.
A may win.
If the goal is:
Increase atmosphere and tension.
B may win.
So Writer does not simply encode the candidate.
During training it constructs:
context embedding + candidate embedding
for each side.
If:
[ g = \text{embedding(goal)} ]
[ a = \text{embedding(candidate A)} ]
[ b = \text{embedding(candidate B)} ]
then the actual model inputs are:
[ x_A = [g \Vert a] ]
and:
[ x_B = [g \Vert b] ]
The architecture therefore becomes:
┌──────────────┐
goal ──┐ │ │
├─ concatenate ─→ │
A ─────┘ │ shared │──→ h_A ──┐
│ encoder │ │
goal ──┐ │ │ ├──→ comparator
├─ concatenate ─→ │ │
B ─────┘ │ │──→ h_B ──┘
└──────────────┘
Now the question is not:
Is A better than B?
It is:
Given this goal, is A better than B?
That is a much more useful model.
One Model Per Dimension
Writer also separates different kinds of preference.
For example:
quality
clarity
simplicity
These are related.
But they are not identical.
A sentence can become clearer while becoming less interesting.
It can become simpler while losing precision.
It can become stylistically better while becoming less concise.
So the training service can build a separate preference model for each dimension:
quality ranker
clarity ranker
simplicity ranker
Conceptually:
┌── quality model ────→ prefers A
A vs B ──────┼── clarity model ────→ prefers B
└── simplicity model ─→ prefers B
This is useful because disagreement is information.
A single scalar can hide the trade-off.
Multiple preference dimensions expose it.
Training the Ranker
Suppose our training pair says:
goal:
Make the sentence clearer.
A:
The model predicts the next token.
B:
The system engages in predictive generation
regarding the subsequently occurring token.
preferred:
A
The model produces a logit:
[ z ]
We convert this into a probability using the sigmoid:
[ p(A > B) = \sigma(z) ]
If A is preferred, our target is:
[ y = 1 ]
If B is preferred:
[ y = 0 ]
We can therefore train using:
criterion = nn.BCEWithLogitsLoss()
The loss asks the network to make the logit positive when A should win and negative when B should win.
This is still ordinary neural-network training:
forward pass
↓
preference logit
↓
loss
↓
backpropagation
↓
optimizer.step()
The unusual part is not gradient descent.
The unusual part is the meaning of the label.
Is This Really Contrastive Learning?
Writer calls this model a contrastive ranker.
That name is useful, but we should be precise about what it means.
There are specific machine-learning objectives usually associated with contrastive learning:
contrastive loss
triplet loss
InfoNCE
Our ranker does not use those losses.
It uses:
BCEWithLogitsLoss
So technically this is better described as a:
pairwise neural preference ranker with a shared encoder.
The contrastive part is in the training structure.
The model learns by seeing contrasts:
A versus B
preferred versus rejected
better versus worse
rather than by receiving an isolated absolute label.
This distinction matters because names should not obscure mechanisms.
There Is a Problem
Our architecture contains a subtle flaw.
Look carefully at the comparator input:
combined = torch.cat([feat_a, feat_b], dim=1)
The model sees:
[A features | B features]
If we reverse the candidates, it sees:
[B features | A features]
Those are different vectors.
Nothing in the architecture itself guarantees:
[ C(A,B) = -C(B,A) ]
But logically we would like something close to that.
If the model strongly says:
A > B
then reversing the pair should give:
B < A
A badly trained network could instead learn something ridiculous:
the left candidate tends to win
This is a positional shortcut.
Writer attacks this problem twice.
First Defence: Reverse Every Training Pair
For every training example:
(A, B) → A wins
Writer also trains:
(B, A) → B loses
So one source preference becomes two training examples.
In tensor form:
input_a, input_b, 1
input_b, input_a, 0
If the original preference says A wins:
X_a.append(input_a)
X_b.append(input_b)
y.append(1.0)
then Writer immediately adds:
X_a.append(input_b)
X_b.append(input_a)
y.append(0.0)
This is a small change with a large consequence.
The network can no longer succeed by learning:
left side = preferred
because every preferred candidate appears on both sides.
This is symmetric pair augmentation.
The Dataset Doubles
If we start with:
100 preference pairs
the trainer sees:
200 ordered training examples
But we have not created 100 new judgments.
We have encoded the symmetry already implied by the original judgments.
This is an important distinction:
source pairs = 100
training pairs = 200
independent evidence = 100
Data augmentation does not create new evidence.
It expresses an invariant more clearly to the model.
Second Defence: Ask the Model Both Ways at Inference
Writer does something even more interesting when comparing candidates.
It computes:
[ z_{AB} = C(A,B) ]
and:
[ z_{BA} = C(B,A) ]
Then:
[ p_{AB} = \sigma(z_{AB}) ]
and:
[ p_{BA} = \sigma(z_{BA}) ]
Ideally:
[ p_{AB} \approx 1 - p_{BA} ]
But neural networks are imperfect.
So instead of trusting either direction alone, Writer calculates:
[ \text{preference}(A)
\frac{ p_{AB} + (1-p_{BA}) }{2} ]
This is a beautiful little trick.
Suppose:
P(A beats B) = 0.80
P(B beats A) = 0.30
The second prediction implies:
P(A beats B) = 1 - 0.30 = 0.70
So we combine them:
[ (0.80 + 0.70)/2 = 0.75 ]
and obtain:
preference A = 0.75
preference B = 0.25
The model is allowed to be slightly inconsistent.
The scoring system corrects for some of that inconsistency.
Architecture and Inference Can Enforce Different Things
This reveals a useful model-design principle.
We could attempt to construct an architecture that mathematically guarantees antisymmetry.
For example, we could learn a scalar utility:
[ s(x) ]
and define:
[ C(A,B) = s(A) - s(B) ]
Then automatically:
[ C(B,A) = -C(A,B) ]
But Writer’s model is more flexible.
It learns:
[ C(A,B) ]
directly from both representations.
That comparator can potentially learn relationships between candidates that are not expressible as the difference between two independent scalar scores.
The cost is that symmetry is no longer guaranteed.
Writer compensates using:
training augmentation
+
bidirectional inference
This is a recurring engineering choice:
Do we put the constraint into the architecture, the training data, the inference procedure, or some combination of all three?
Ranking Is Not Scoring
At this point the model can answer:
A or B?
But Writer frequently needs another operation:
How good is A?
These are different questions.
A pairwise model naturally produces:
[ A > B ]
It does not automatically tell us:
[ A = 0.82 ]
This is one of the biggest conceptual differences between MR.Q and the preference ranker.
MR.Q directly learns something like:
(context, response) → score
The ranker naturally learns:
(context, A, B) → preference
So how can we use a ranker as a scorer?
Introduce a Baseline
Writer uses a clever solution.
Give the ranker a fixed reference sentence.
For example:
This sentence is weak, generic, unclear, or stylistically worse.
Now instead of asking:
Is this sentence good?
we ask:
Is this sentence better than the baseline?
The pair becomes:
candidate
vs
baseline
The model gives us a logit.
A very weak candidate may barely beat the baseline.
A strong candidate may beat it decisively.
Now we have something that behaves much more like a scalar.
candidate 1 ──→ barely beats baseline
candidate 2 ──→ clearly beats baseline
candidate 3 ──→ overwhelmingly beats baseline
The baseline acts as an anchor.
From Relative Preference to an Absolute-ish Score
Suppose we also possess some examples with absolute values.
We can collect:
ranker logit known score
---------------------------
-0.8 0.21
0.1 0.48
0.9 0.71
1.8 0.88
Then train a small calibration model that maps:
[ \text{ranker logit} \rightarrow \text{absolute score} ]
Writer calls this component a RegressionTuner.
The entire scoring pipeline becomes:
goal
candidate
baseline
↓
embeddings
↓
pairwise ranker
↓
raw preference logit
↓
calibration
↓
0-to-1 score
This is not magically creating an objective measurement of quality.
It is anchoring a relative model to an externally defined scale.
That distinction is important.
The ranker still fundamentally understands comparisons.
Calibration gives us a convenient coordinate system.
This Is Surprisingly General
This baseline trick appears in many forms.
Suppose we know:
A > reference
B > reference by more
C < reference
We have started constructing an ordering around an anchor.
The same general idea appears in:
- rating systems
- psychometrics
- preference learning
- reward modelling
- search ranking
- recommendation systems
- human-feedback training
The model may learn comparatively while the surrounding system exposes a scalar interface.
Why Writer Uses This So Much
Writer continuously encounters pairs.
An original sentence and a revision:
original
revised
An accepted edit and a rejected alternative:
accepted
rejected
Two candidate rewrites:
candidate A
candidate B
A human decision therefore naturally creates training data:
preferred
not preferred
We do not need to interrupt the human and ask:
Please assign this revision a clarity value of exactly 0.734.
The interaction itself reveals preference.
That is powerful.
Normal product activity can generate training signals.
Preferences Can Be About the User, Not Universal Truth
There is another subtle point here.
Suppose Writer observes:
original:
The implementation provides a robust and comprehensive mechanism
for performing validation.
accepted:
The implementation validates the result.
The lesson is not necessarily:
Short sentences are objectively better.
The lesson may instead be:
Under this goal, for this writing system, this revision was preferred.
That is why context matters so much.
Preference models are especially useful when the desired behaviour is:
situated
contextual
subjective
goal-dependent
rather than a universal physical quantity.
The Ranker Can Still Fail
A preference model is not automatically trustworthy.
Suppose it sees:
A > B
B > C
C > A
We have produced a cycle.
There is no single scalar ordering that satisfies all three preferences.
Human preference data can contain these inconsistencies.
Different contexts can produce them legitimately.
Noise can produce them accidentally.
A flexible pairwise comparator can learn local relationships without guaranteeing a globally consistent ranking.
So we should distinguish:
pairwise preference
from:
global utility function
They are not automatically the same thing.
Confidence Is Not Quality
Writer’s scoring layer records another useful quantity:
margin = |score_a - score_b|
If:
A = 0.51
B = 0.49
the ranker has only a slight preference.
If:
A = 0.91
B = 0.09
the separation is much clearer.
But the margin does not mean A has high quality.
Consider:
terrible sentence A
even worse sentence B
The model may confidently prefer A.
That tells us:
A is clearly preferable to B
not:
A is excellent
This distinction is easy to lose when pairwise systems are wrapped in numerical APIs.
Preference strength and absolute quality are different variables.
Model Health Matters Too
There is one final systems lesson in Writer’s implementation.
Suppose no ranker model loaded successfully.
Or the requested dimension does not exist.
A badly designed scoring system might still return:
0.5
and downstream code might interpret that as:
the model believes these candidates are equal
But those are not the same thing.
There is an enormous difference between:
the model evaluated A and B and decided they tie
and:
there was no functioning model
Writer therefore carries health information alongside the preference result.
Conceptually:
winner
score A
score B
used dimensions
loaded dimensions
health
load errors
This is a useful general principle:
Model failure should remain distinguishable from model uncertainty.
From Prediction to Explanation
Writer then takes another step.
The pairwise decision can be converted into a small causal score graph.
Instead of retaining only:
A wins
the system can preserve:
pairwise preference
pairwise margin
ranker health
quality contribution
clarity contribution
simplicity contribution
external penalties
So the model’s output becomes part of a larger evidence structure.
This is no longer just:
neural network → number
It becomes:
neural network
↓
structured decision
↓
contributors
↓
evidence
↓
downstream reasoning
This is worth noticing because production models almost never live alone.
The neural network may be only twenty lines long.
The useful system around it can be much larger.
MR.Q Versus the Preference Ranker
We can now see two different approaches to the same broad problem.
MR.Q
context
response
↓
model
↓
absolute scalar
Training data:
response → 0.82
Preference ranker
context
A
B
↓
model
↓
A > B
Training data:
A preferred to B
Neither architecture is universally better.
They assume different kinds of supervision.
Use an absolute scorer when you genuinely have meaningful absolute targets.
Use a preference ranker when comparative judgments are more natural and reliable.
The Training Signal Should Influence the Architecture
This is perhaps the biggest lesson of the chapter.
We often begin model design by asking:
Which neural network architecture should I use?
But there is an earlier question:
What information do I actually possess?
If your data naturally looks like:
input → class
build a classifier.
If it looks like:
input → number
build a regressor.
If it looks like:
context + action → reward
build something like MR.Q.
If it looks like:
A was preferred to B
build a ranker.
The shape of the supervision should influence the shape of the model.
A Tiny Preference Ranker From Scratch
We can reduce the whole idea to a very small PyTorch example.
import torch
import torch.nn as nn
class TinyRanker(nn.Module):
def __init__(self, input_dim=16, hidden_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(),
)
self.comparator = nn.Linear(hidden_dim * 2, 1)
def forward(self, a, b):
ha = self.encoder(a)
hb = self.encoder(b)
pair = torch.cat([ha, hb], dim=1)
return self.comparator(pair).squeeze(1)
Create some fake candidates:
a = torch.randn(8, 16)
b = torch.randn(8, 16)
Suppose A wins in some rows and B wins in others:
labels = torch.tensor([
1, 0, 1, 1,
0, 0, 1, 0,
], dtype=torch.float32)
Train exactly as before:
model = TinyRanker()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()
logits = model(a, b)
loss = criterion(logits, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Nothing exotic has happened.
We still have:
Linear
ReLU
concatenate
Linear
loss
backpropagation
Adam
The novelty comes from how those familiar pieces are arranged around a different question.
The Recursive Decomposition Still Works
This brings us back to the theme of the entire book.
A preference ranker may sound like a specialized machine-learning architecture.
Decompose it:
PreferenceRanker
↓
shared encoder + comparator
↓
Linear + ReLU + Dropout + Linear
↓
matrix multiplication + bias
The training system sounds more complicated:
ContrastiveRankerTrainer
Decompose it:
preference pairs
↓
embeddings
↓
context concatenation
↓
symmetric pair augmentation
↓
standardization
↓
BCEWithLogitsLoss
↓
Adam
The scoring system sounds more complicated again:
ContrastiveRankerScorer
Decompose it:
A vs B
B vs A
↓
two probabilities
↓
symmetry correction
↓
dimension aggregation
And absolute scoring becomes:
candidate vs baseline
↓
preference logit
↓
calibration
↓
score
At every level, the intimidating name disappears when we recursively open the box.
The Bigger Lesson
There is a deeper reason I wanted to include this model.
Most introductory machine learning teaches us to think in labels:
cat
dog
0.73
positive
negative
But a huge amount of useful information in the real world arrives as choice.
Someone selected this result instead of that one.
They accepted this edit.
Rejected another.
Clicked one result.
Ignored another.
Preferred answer A.
Reverted answer B.
Those interactions do not necessarily tell us what absolute score anything deserves.
They tell us something simpler:
[ A > B ]
And sometimes that is enough.
A preference ranker turns those local choices into a learnable model.
It does not need us to pretend that subjective judgments are precise measurements.
It starts from the information we actually have.
That makes it one of the simplest models in this book architecturally.
And one of the most useful conceptually.
Because sometimes the right question is not:
How good is this?
It is simply:
Which one would you keep?
