Semantic Failure Analysis and Hard Negatives

A user requests a battery-replacement procedure. The first-ranked passage says that the notes do not describe battery replacement. It matches the topic but cannot supply the requested steps. Both lexical and dense ranking put that passage first: a strong match to the question is not necessarily evidence that can answer it.

Changing from lexical to dense ranking does not correct this mistake. Neither does filtering for eligibility: the negative statement is permitted. Filters, Scope, and Freshness showed that a candidate can pass the policy gate and still be insufficient for the task.

We need to name the failure more precisely than “semantic search failed”. Did the method miss the required evidence, favour a misleading neighbour, lose an essential condition or select evidence that could not answer the need? The diagnosis should tell us what a later stage would have to change.

Construct a challenge before inspecting its outcome

The shared fictional collection contains four deliberately misleading neighbours. An R71 procedure looks like the scoped R17 procedure but specifies a different identifier and duration. An operations note contradicts the accepted laboratory-key statement. A cache overview discusses the right subject without giving the command. And the existing battery note mentions replacement only to say that no procedure is described. Each can look useful to a matcher for a different reason.

These negatives were constructed for the experiment, not sampled from real user failures. Their source records and required evidence bundles were declared before the model comparisons. We call them “hard” only where the observed rankings justify it: some appear near or above accepted evidence. We do not assume that every negative is difficult for every method or query.

The accepted source is part of the judgement, not something the retriever establishes. For the laboratory-key need, the declared sufficient evidence comes from a particular source and version. That fictional authority has not been verified independently, and its label is not proof of truth. The test asks whether retrieval preserves the authored source requirement.

The eight protocol-held-out questions are rewordings of eight development needs, not eight new evidence problems. Seven retain exactly the same non-empty anchor bundles; the eighth retains the same empty-bundle no-answer condition. We therefore call this the reworded-query slice. Its recorded split tests changed wording against already constructed needs, not generalisation to unseen documents, new evidence requirements or independently authored questions. Agreement between the development and reworded-query results is not two independent confirmations. The original split identifiers remain in the evidence records.

The same implementing agent wrote and could inspect the judgements, so neither split is blind or independently adjudicated. The comparison did not mine negatives from held-out output for training or change queries, anchors, models, chunk sizes or LSH settings after evaluation. A future tuning study would need newly constructed evidence needs, independent adjudication and a fresh final test. We will not reuse these now-visible outcomes as if they were unseen evaluation data.

Locate the lost obligation

In training research, “hard negatives” often refer to candidates used to teach a retriever, not merely to diagnose it. ANCE mines corpus candidates through an ANN index during dense-retriever training and refreshes representations as the model changes. RocketQA examines the false-negative risk: an unlabelled high-ranking passage may actually be relevant. Our fixed-model challenge instead uses declared source-evidence requirements. We have neither mined training negatives nor implemented denoising. Turning these examples into training data would require checked labels and a new evaluation, not reuse of the held-out results.

The frozen traces let us classify failures by the obligation they leave unsatisfied. The table separates what was actually observed from the next experiment that observation might justify. Its proposed remedies are hypotheses, not results.

None of the proposed remedies in the table was tested as a remedy in this chapter. The location of the loss constrains what could help: a reranker cannot promote a required passage that never entered its pool, and a larger pool cannot recover words omitted from every indexed representation by encoder truncation.

The map below places each recorded Part II failure at the stage where it occurred. Read it as a location list: it says which later stage could possibly repair a loss, not which method is generally better.

Read across the stages, each answer rules out a class of remedy rather than naming a culprit: the same empty result can be produced by a missing candidate, a withheld permission, a late gate or an unsatisfied bundle.

The long-section case establishes two observations: the selector instruction was absent from the encoder input, and the retained dense ranking missed its required evidence. It does not separate the effect of that omission from possible dilution by unrelated text. Dilution remains a hypothesis; neither a larger input limit nor another pooling policy has been verified as a repair. Exploratory work outside the frozen comparison is not a book result.

Compare failures at the same declared cutoff

With section retrieval units and a three-result cutoff, exact dense search covers seven of nine answerable development needs and six of seven answerable reworded needs. BM25 covers eight of nine and seven of seven respectively. Those totals do not tell the whole story. For the reworded scope request, dense ranks the accepted scoped procedure first while BM25 ranks it second. Both succeed by three results, but they deliver the useful evidence in a different order.

For the development paraphrase, neither exact BM25 nor exact dense search covers the required bundle within three results under any segmentation scheme. Approximate window search does succeed, but only because its candidate filter removes higher-scoring irrelevant neighbours. That favourable accident shows why agreement with exact neighbours and success against evidence judgements are different measurements.

The reworded R71 request exposes a different boundary. Exact section search puts the accepted twenty-second R71 instruction first. Exact block search selects headings and older reset instructions without covering the required R71 action. With windows, BM25 covers that instruction across two fragments, neither of which is sufficient on its own. The same information need therefore yields different coverage depending on which source spans survive selection.

Use the retained query traces to inspect these cases:

python -B experiments/part2-semantic/run.py --query s-d08
python -B experiments/part2-semantic/run.py --query s-h07
python -B experiments/part2-semantic/run.py --query s-d05

Every selected result resolves through its segmentation-specific manifest to an exact span of retained parent text. That lets us inspect which evidence was chosen rather than interpret the score in isolation. BM25 scores and cosine similarities have different meanings and scales; comparing their numbers as probabilities would obscure, not explain, these failures.

Duplicates and contradictions require different treatment

For the lamp-status need, several occurrences in the specification and blog are explicitly acceptable alternatives. One accepted occurrence is enough for query-level bundle success; further copies consume result positions without increasing that success value. We preserve each occurrence’s source and passage identity, even where the text is identical, so the reader can still inspect its origin.

The conflicting laboratory-key statement is not a duplicate of an accepted answer. Its similar wording does not satisfy the declared source requirement. A result set containing both the accepted laboratory passage and the contradictory operations passage can still meet structural bundle coverage, but that metric does not tell us whether a consumer would resolve the conflict correctly. Merging them would erase evidence of disagreement. Deduplication, source authority and contradiction handling require distinct rules, not a shared similarity threshold.

The same data boundary applies to the hostile imported quotation. A request about that quotation may legitimately retrieve it as source text; retrieval does not authorise a consumer to follow its instructions. No downstream model was run, so this experiment measures neither prompt-injection success nor resistance.

Keep no-answer outcomes outside the success average

Each query split contains one battery-procedure need for which the collection declares no sufficient evidence bundle. All three methods return candidates under every segmentation scheme, and the negative battery note ranks first. Bundle success is undefined for these needs, not zero: there is no accepted bundle to recover. We record candidate return separately rather than adding an artificial failure to the answerable-query average or treating the first result as a correct answer.

The negative note might support an answer to a different question, such as whether these notes provide a replacement procedure. It cannot satisfy the current request for the procedure itself. Changing the judgement after seeing the ranking would erase that distinction. A separate evaluation could test evidence-supported refusal, with its own references and consumer outcomes. Here an empty candidate list and a non-empty one remain observations, not automatic verdicts about answerability.

Not finding sufficient evidence, refusing evidence under policy and missing a permitted neighbour through approximation are three different failure paths. Any may produce an empty or inadequate result, but each calls for a different diagnosis. The public response need not reveal protected reasons; an authorised offline audit can still identify which boundary failed.

Define what the next stage must prove

Part II has now added a real pinned encoder, an exact nearest-neighbour reference, an approximate candidate pool and a current-view eligibility gate to the lexical baseline. Together they make several losses inspectable: source spans can be omitted, wrong or conflicting neighbours can rank highly, approximation can discard candidates, and a stale index can expose a permission risk. These are diagnostic results from a small, authored collection, not evidence of general performance. Resident memory, production security, context admission, evidence use and downstream helpfulness remain not measured.

Candidate Generation as a Pipeline takes these findings into Part III. It will track which lexical and dense candidates reach a shared pool, vary candidate breadth separately from ranking, and preserve each source of evidence. Whether combining those lists helps remains an experimental question—not a result established by Part II.

Before adding another ranking stage, we must first answer a simpler question: did the required source evidence reach the candidate pool? If it did not, no later ordering rule can select it.

Capstone checkpoint: execute the representation boundary

python -B -m examples.capstone.milestones 2 --summary

The capstone milestone extends the teaching collection with actual pinned local MiniLM vectors. Its summary displays lexical and dense ordering, model setup time and the restricted lexical control; it does not label semantic search a success merely because it ran. The full trace is saved in .artifacts/capstone/runtime/milestones/02.json, and omitting --summary retains the machine-readable status output. Stale source data is excluded from the eligible index. If the pinned model or dependencies are unavailable, the dense capability reports an error rather than substituting fabricated vectors. This integration check does not alter the frozen Part II results or establish better downstream answers.

References and further reading

Background and extensions for this chapter:

• Xiong et al. (2020). Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. arXiv preprint, v2. Motivates mining difficult training negatives from the searchable corpus.
• Qu et al. (2021). RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. NAACL-HLT. Examines denoised hard negatives and the risk of unlabelled positives.
• Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks. Extends failure analysis beyond one training domain or narrow query collection.

Related chapters

• Hard Negatives. Extends hard-negative analysis to selection rules and false negatives.