Example

Two embeddings in, one number out, and what gives the number meaning

Train the chapter's pairwise quality model on a synthetic task, then check it against a cosine baseline.

An architecture that maps a pair of embeddings to a scalar says nothing about what that scalar means. The meaning comes from the objective, the data and the evaluation, not from the name of the output.

This program trains the chapter’s pairwise model on a synthetic task, then asks the chapter’s question: is the network better than the cosine similarity that needs no training?

Run it

Save this as mrq_quality_model.py and run it with Python 3.11 or later. It needs PyTorch: pip install torch, then python mrq_quality_model.py.

import torch
import torch.nn.functional as F
from torch import nn


class PairEncoder(nn.Module):
    def __init__(self, embedding_dim, hidden_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(embedding_dim * 2, hidden_dim), nn.ReLU(), nn.LayerNorm(hidden_dim),
            nn.Linear(hidden_dim, hidden_dim), nn.ReLU())

    def forward(self, context_emb, response_emb):
        assert context_emb.shape == response_emb.shape
        return self.net(torch.cat([context_emb, response_emb], dim=-1))


class ScalarPredictor(nn.Module):
    def __init__(self, hidden_dim):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(hidden_dim, hidden_dim // 2), nn.ReLU(), nn.Linear(hidden_dim // 2, 1))

    def forward(self, z):
        return self.net(z).squeeze(-1)


class MRQStyleModel(nn.Module):
    def __init__(self, embedding_dim=32, hidden_dim=64):
        super().__init__()
        self.encoder = PairEncoder(embedding_dim, hidden_dim)
        self.predictor = ScalarPredictor(hidden_dim)

    def forward(self, context_emb, response_emb):
        return self.predictor(self.encoder(context_emb, response_emb))


def pairwise_logistic_loss(q_preferred, q_rejected):
    return F.softplus(-(q_preferred - q_rejected)).mean()


torch.manual_seed(7)
model = MRQStyleModel(32, 64)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for step in range(1000):
    context = torch.randn(128, 32)
    preferred = context + 0.2 * torch.randn(128, 32)
    rejected = torch.randn(128, 32)
    q_good = model(context, preferred)
    q_bad = model(context, rejected)
    loss = pairwise_logistic_loss(q_good, q_bad)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()
    if step % 250 == 0:
        with torch.no_grad():
            acc = (q_good > q_bad).float().mean()
        print(f"step {step:4d} loss={loss.item():.4f} ranking_acc={acc.item():.3f}")

with torch.no_grad():
    context = torch.randn(512, 32)
    preferred = context + 0.2 * torch.randn(512, 32)
    rejected = torch.randn(512, 32)
    q_good, q_bad = model(context, preferred), model(context, rejected)
    cos_good = F.cosine_similarity(context, preferred, dim=-1)
    cos_bad = F.cosine_similarity(context, rejected, dim=-1)
    single = model(context[:1], preferred[:1])
print(f"held-out ranking accuracy: model {(q_good > q_bad).float().mean():.3f}  cosine baseline {(cos_good > cos_bad).float().mean():.3f}")
print(f"mean score margin:        model {(q_good - q_bad).mean():.3f}  cosine baseline {(cos_good - cos_bad).mean():.3f}")
print(f"a batch of one still gives a score per example: {tuple(single.shape)}")
print(f"trainable parameters: {sum(p.numel() for p in model.parameters()):,d}")

Expected outcome

step    0 loss=0.6904 ranking_acc=0.547
step  250 loss=0.0011 ranking_acc=1.000
step  500 loss=0.0002 ranking_acc=1.000
step  750 loss=0.0000 ranking_acc=1.000
held-out ranking accuracy: model 1.000  cosine baseline 1.000
mean score margin:        model 12.988  cosine baseline 0.976
a batch of one still gives a score per example: (1,)
trainable parameters: 10,561

The objective gives the scalar its meaning

The loss never asks the model to recover a number. It asks for an ordering: the preferred score minus the rejected score should be positive.

def pairwise_logistic_loss(preferred_score, rejected_score):
    margin = preferred_score - rejected_score
    return torch.nn.functional.softplus(-margin).mean()

Trained with mean-squared error, the same architecture would learn a different quantity with the same name.

The held-out numbers are worth sitting with. The model reached ranking accuracy 1.000 on pairs it never saw, and so did the cosine baseline. The training bought a bigger margin, not a better ordering.

Mechanism and limitations

Documented: the chapter offers three objectives for the same architecture, gives the ranking loss as softplus(-margin), trains with AdamW for a thousand steps, measures ranking accuracy, and warns that cosine similarity may solve much of the task.

Observed: the run above. The loss fell from 0.6904 to 0.0000 and the training ranking accuracy reached 1.000 by step 250. On 512 held-out pairs the model and the cosine baseline both reached 1.000 ranking accuracy, with mean margins of 12.988 and 0.976. A batch of one gave one score per example.

Not shown. The program composes the chapter’s fragments into one run and trains on synthetic embeddings, so it says nothing about real text. Both the network and the baseline are perfect here, so the run does not separate them. Nothing tests calibration.

Canonical source , the chapter this example composes. Read the chapter .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.