Two embeddings in, one number out, and what gives the number meaning
Train the chapter's pairwise quality model on a synthetic task, then check it against a cosine baseline.
An architecture that maps a pair of embeddings to a scalar says nothing about what that scalar means. The meaning comes from the objective, the data and the evaluation, not from the name of the output.
This program trains the chapter’s pairwise model on a synthetic task, then asks the chapter’s question: is the network better than the cosine similarity that needs no training?
Run it
Save this as mrq_quality_model.py and run it with Python 3.11 or later. It
needs PyTorch: pip install torch, then python mrq_quality_model.py.
import torch
import torch.nn.functional as F
from torch import nn
class PairEncoder(nn.Module):
def __init__(self, embedding_dim, hidden_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(embedding_dim * 2, hidden_dim), nn.ReLU(), nn.LayerNorm(hidden_dim),
nn.Linear(hidden_dim, hidden_dim), nn.ReLU())
def forward(self, context_emb, response_emb):
assert context_emb.shape == response_emb.shape
return self.net(torch.cat([context_emb, response_emb], dim=-1))
class ScalarPredictor(nn.Module):
def __init__(self, hidden_dim):
super().__init__()
self.net = nn.Sequential(nn.Linear(hidden_dim, hidden_dim // 2), nn.ReLU(), nn.Linear(hidden_dim // 2, 1))
def forward(self, z):
return self.net(z).squeeze(-1)
class MRQStyleModel(nn.Module):
def __init__(self, embedding_dim=32, hidden_dim=64):
super().__init__()
self.encoder = PairEncoder(embedding_dim, hidden_dim)
self.predictor = ScalarPredictor(hidden_dim)
def forward(self, context_emb, response_emb):
return self.predictor(self.encoder(context_emb, response_emb))
def pairwise_logistic_loss(q_preferred, q_rejected):
return F.softplus(-(q_preferred - q_rejected)).mean()
torch.manual_seed(7)
model = MRQStyleModel(32, 64)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
for step in range(1000):
context = torch.randn(128, 32)
preferred = context + 0.2 * torch.randn(128, 32)
rejected = torch.randn(128, 32)
q_good = model(context, preferred)
q_bad = model(context, rejected)
loss = pairwise_logistic_loss(q_good, q_bad)
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()
if step % 250 == 0:
with torch.no_grad():
acc = (q_good > q_bad).float().mean()
print(f"step {step:4d} loss={loss.item():.4f} ranking_acc={acc.item():.3f}")
with torch.no_grad():
context = torch.randn(512, 32)
preferred = context + 0.2 * torch.randn(512, 32)
rejected = torch.randn(512, 32)
q_good, q_bad = model(context, preferred), model(context, rejected)
cos_good = F.cosine_similarity(context, preferred, dim=-1)
cos_bad = F.cosine_similarity(context, rejected, dim=-1)
single = model(context[:1], preferred[:1])
print(f"held-out ranking accuracy: model {(q_good > q_bad).float().mean():.3f} cosine baseline {(cos_good > cos_bad).float().mean():.3f}")
print(f"mean score margin: model {(q_good - q_bad).mean():.3f} cosine baseline {(cos_good - cos_bad).mean():.3f}")
print(f"a batch of one still gives a score per example: {tuple(single.shape)}")
print(f"trainable parameters: {sum(p.numel() for p in model.parameters()):,d}")Expected outcome
step 0 loss=0.6904 ranking_acc=0.547
step 250 loss=0.0011 ranking_acc=1.000
step 500 loss=0.0002 ranking_acc=1.000
step 750 loss=0.0000 ranking_acc=1.000
held-out ranking accuracy: model 1.000 cosine baseline 1.000
mean score margin: model 12.988 cosine baseline 0.976
a batch of one still gives a score per example: (1,)
trainable parameters: 10,561The objective gives the scalar its meaning
The loss never asks the model to recover a number. It asks for an ordering: the preferred score minus the rejected score should be positive.
def pairwise_logistic_loss(preferred_score, rejected_score):
margin = preferred_score - rejected_score
return torch.nn.functional.softplus(-margin).mean()Trained with mean-squared error, the same architecture would learn a different quantity with the same name.
The held-out numbers are worth sitting with. The model reached ranking
accuracy 1.000 on pairs it never saw, and so did the cosine baseline. The
training bought a bigger margin, not a better ordering.
Mechanism and limitations
Documented: the chapter offers three objectives for the same
architecture, gives the ranking loss as softplus(-margin), trains with
AdamW for a thousand steps, measures ranking accuracy, and warns that cosine
similarity may solve much of the task.
Observed: the run above. The loss fell from 0.6904 to 0.0000 and the
training ranking accuracy reached 1.000 by step 250. On 512 held-out pairs
the model and the cosine baseline both reached 1.000 ranking accuracy, with
mean margins of 12.988 and 0.976. A batch of one gave one score per
example.
Not shown. The program composes the chapter’s fragments into one run and trains on synthetic embeddings, so it says nothing about real text. Both the network and the baseline are perfect here, so the run does not separate them. Nothing tests calibration.
Canonical source , the chapter this example composes. Read the chapter .
Understand this example
Copy this prompt into your AI tool. No code runs here.
Apply this example
Copy this prompt into your AI tool. No code runs here.