Part III — Retrieval Is an Experiment

Two benchmarks, same model, opposite verdicts

Benchmark E (easy negatives):
  positive: the correct answer
  negatives: 20 passages drawn at random from the corpus
  → model scores Recall@1 = 0.94

Benchmark H (hard negatives):
  positive: the correct answer
  negatives: the 20 passages most similar to the query that are NOT correct
  → same model scores Recall@1 = 0.55

Nothing changed but the negatives. The easy benchmark asked “can the model tell the answer from an unrelated paragraph?” The hard benchmark asked “can it tell the answer from the 20 things that look most like the answer?” Those are different questions, and only the second is the one production faces.

What distinction is a benchmark actually testing — and are its negatives hard enough to test the distinction you care about?

How to mine hard negatives

• In-model (self-adversarial). Embed everything with the model under test; for each query, take its top-k non-answers as negatives. Fast, and directly targets the model’s blind spots. Risk: label noise — some “non-answers” are actually relevant (false negatives).
• Cross-model. Use a different, strong model to find candidates that are similar but labeled non-relevant. Reduces the self-fulfilling aspect.
• Structured perturbation. Generate negatives by transformation: negate the positive, swap its entities, shift its dates, weaken its quantifiers, replace its key relation. These target specific capabilities.
• Lexical-overlap matched. Select non-answers with high BM25 / token overlap with the query. Targets the “keyword trap.”
• Entity-matched. Non-answers sharing the query’s named entities.

Good hard-negative sets stratify by type so you learn which distinction fails, not just that the score dropped.

The false-negative problem

If you mine negatives as “top similar, not labeled positive,” some will be unlabeled positives — genuinely relevant passages your labels missed. Training or evaluating on those punishes the model for being right. Mitigations: a margin (skip the top 1–2 as likely positives), human spot-checks, or a strong cross-model relevance filter. Report the estimated false-negative rate of your hard-negative set; it bounds how much of the score drop is real.

What hard negatives are for

• Evaluation: they reveal the operating regime the model will actually face. An easy-negative Recall@1 of 0.94 tells you almost nothing.
• Training: contrastive learning with hard negatives (in-batch, then mined) is how modern retrieval models get good — the gradient signal from an easy negative is near zero.
• Diagnosis: stratified hard negatives localize the failure to a capability (polarity, role, time, quantity).

Demonstration: margin collapse on RELATE

MEASURED on RELATE v0.1, Wave 1 row 1.8 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/margin-collapse.json. Model: all-mpnet-base-v2.

For the 269 RELATE queries, compute the margin: cosine(query, correct) − cosine(query, hardest negative in the set).

negative set                    mean margin   Recall@1 vs that set
random (same-domain)               +0.47             1.00
in-model top-5 non-answers         +0.21             0.98
structured perturbations           +0.09             0.93
BM25 lexical-overlap-matched       +0.06             0.76

structured margin by relation:  negation +0.14   temporal-mismatch +0.10   relation-swap +0.03

MEASURED: the comfortable +0.47 margin from random negatives is almost entirely “this passage is not about the topic.” Against negatives that are about the topic it collapses by 5–8×; against a role-reversed relation-swap it is +0.03 — statistically present, operationally gone. On RELATE v0.1 with this model the margin does not go negative (the ranking holds ~93% of the time against structured negatives), but its resolution for the distinction that matters is down at the noise floor, which is where calibration (Chapter 14) fails.

What this chapter establishes and what it does not

Establishes: benchmark difficulty is set by the negatives; easy negatives measure topical separation, hard negatives measure the target distinction; methods to mine and stratify hard negatives; the false-negative hazard and how to bound it; margin as the quantity that collapses.

Does not establish: that a model is “bad” (it may be excellent at topical retrieval, which is sometimes all you need), or a universal hard-negative recipe. It establishes that any retrieval claim must state the negative distribution it was measured against.

Lab 11: build a hard-negative benchmark and watch the margin fall

PROPOSED, not executed.

Setup. 200 queries with labeled positives. One model under test, one auxiliary model.

Task.

1. Build four negative sets: random, BM25-matched, in-model top-5, structured (≥2 perturbation types).
2. For each, compute mean margin, Recall@1, Recall@10, MRR.
3. Estimate the false-negative rate of the in-model set (spot-check 30).
4. Stratify the structured results by perturbation type.

Success criterion. One number — the Recall@1 gap between random and hardest negatives — and one sentence naming the capability that collapses first for your model.

Companion component: the negative-set descriptor

negative_set_descriptor:
  mining_method:      [random | bm25 | in_model | cross_model | structured]
  strata:             {negation: n, relation_swap: n, temporal: n, ...}
  est_false_negative_rate: float
  mean_margin:        float
  note:               "scores below are meaningless without this block"

The Observatory refuses to display a retrieval quality number without an attached negative-set descriptor, and defaults its evaluation to mined hard negatives.

Failure modes

• Reporting easy-negative scores as “retrieval quality.” They measure topical separation only.
• Mining hard negatives without a false-negative estimate. Some of your “negatives” are positives; the drop is partly artificial.
• Unstratified hard negatives. You learn the score fell, not what failed.
• Training on noisy hard negatives. Teaches the model to down-rank correct answers.

What this chapter established

• The negatives set the difficulty: easy = topical separation, hard = the target distinction.
• Mining methods: in-model, cross-model, structured perturbation, lexical-matched, entity-matched — and why to stratify.
• The false-negative hazard and how to bound the real portion of a score drop.
• Margin collapse (measured, row 1.8): the +0.47 random-negative margin falls to +0.06 against BM25-lexical negatives and to +0.03 against role-reversed relation-swap — present but operationally gone, not (on RELATE v0.1 with mpnet) actually inverted.
• The negative-set descriptor: no retrieval number is shown without it.

Next

We have pushed retrieval quality down with hard negatives. The next chapter zooms out: retrieval is not one operation but a chain of decisions — representation, candidates, similarity, threshold, ranking, top-k, filters, budget — and changing any link changes the “memory” the system ends up with.