Chapter research

Research: Semantic Failure Analysis and Hard Negatives

A starting point for fresh evidence and thoughtful chapter updates. No research has been recorded against this chapter yet.

Read the chapter

Research status Brief only · not yet researched

Take the contextCopy the chapter research prompt
Investigate the evidenceUse your preferred research AI
Review the changesDecide what belongs in the book

Ready to use

Research prompt

Plain text

Chapter context, research questions, and instructions for a report you can review. Copy it into your preferred research AI.

Download prompt .txt

Full research prompt Current chapter context

You are researching one chapter of an open technical book so that its author can decide, with evidence, whether the chapter needs correcting, clarifying, citing or leaving alone.

== 1. Identity ==
Book:            Retrieval From First Principles
Chapter:         Semantic Failure Analysis and Hard Negatives
Chapter number:  10
Stable id:       retrieval-10-chapter
Chapter URL:     https://programmer.ie/books/retrieval/10-chapter/
Research pack:   https://programmer.ie/research/books/retrieval/10-chapter/
Chapter prose fingerprint (sha256, normalized): d03c00d329b039d0360e3533d9acc25501485ed5da18058fae1a03edc4640330

The chapter text below is an EXCERPT, not the whole chapter.
A complete plain-text snapshot of this exact revision is downloadable at https://programmer.ie/research/books/retrieval/10-chapter/chapter-snapshot.txt
If you cannot fetch it, say so and ask me to paste the chapter. Do not guess at the missing text.

== 2. Chapter text ==

A user requests a battery-replacement procedure. The first-ranked passage says that the notes do not describe battery replacement. It matches the topic but cannot supply the requested steps. Both lexical and dense ranking put that passage first: a strong match to the question is not necessarily evidence that can answer it.
Changing from lexical to dense ranking does not correct this mistake. Neither does filtering for eligibility: the negative statement is permitted. Filters, Scope, and Freshness showed that a candidate can pass the policy gate and still be insufficient for the task.
We need to name the failure more precisely than “semantic search failed”. Did the method miss the required evidence, favour a misleading neighbour, lose an essential condition or select evidence that could not answer the need? The diagnosis should tell us what a later stage would have to change.
Construct a challenge before inspecting its outcome The shared fictional collection contains four deliberately misleading neighbours. An R71 procedure looks like the scoped R17 procedure but specifies a different identifier and duration. An operations note contradicts the accepted laboratory-key statement. A cache overview discusses the right subject without giving the command. And the existing battery note mentions replacement only to say that no procedure is described. Each can look useful to a matcher for a different reason.
These negatives were constructed for the experiment, not sampled from real user failures. Their source records and required evidence bundles were declared before the model comparisons. We call them “hard” only where the observed rankings justify it: some appear near or above accepted evidence. We do not assume that every negative is difficult for every method or query.
The accepted source is part of the judgement, not something the retriever establishes. For the laboratory-key need, the declared sufficient evidence comes from a particular source and version. That fictional authority has not been verified independently, and its label is not proof of truth. The test asks whether retrieval preserves the authored source requirement.
The eight protocol-held-out questions are rewordings of eight development needs, not eight new evidence problems. Seven retain exactly the same non-empty anchor bundles; the eighth retains the same empty-bundle no-answer condition. We therefore call this the reworded-query slice. Its recorded split tests changed wording against already constructed needs, not generalisation to unseen documents, new evidence requirements or independently authored questions. Agreement between the development and reworded-query results is not two independent confirmations. The original split identifiers remain in the evidence records.
The same implementing agent wrote and could inspect the judgements, so neither split is blind or independently adjudicated. The comparison did not mine negatives from held-out output for training or change queries, anchors, models, chunk sizes or LSH settings after evaluation. A future tuning study would need newly constructed evidence needs, independent adjudication and a fresh final test. We will not reuse these now-visible outcomes as if they were unseen evaluation data.
Locate the lost obligation In training research, “hard negatives” often refer to candidates used to teach a retriever, not merely to diagnose it. ANCE mines corpus candidates through an ANN index during dense-retriever training and refreshes representations as the model changes. RocketQA examines the false-negative risk: an unlabelled high-ranking passage may actually be relevant. Our fixed-model challenge instead uses declared source-evidence requirements. We have neither mined training negatives nor implemented denoising. Turning these examples into training data would require checked labels and a new evaluation, not reuse of the held-out results.
The frozen traces let us classify failures by the obligation they leave unsatisfied. The table separates what was actually observed from the next experiment that observation might justify. Its proposed remedies are hypotheses, not results.
Failure What the trace shows Proposed next question Vocabulary mismatch BM25 returns nothing for purge temporary stored data Could a separately evaluated semantic candidate source recover the command? Topical neighbour Exact dense search also misses that command by three results Could a later relevance check distinguish an overview from an executable instruction? Identifier confusion Block-level dense ranking prefers R17 to the requested R71 instruction Would preserving identifier context or combining lexical candidates help? Conflicting evidence The block-level conflicting key assertion outranks the accepted laboratory statement Can declared source requirements survive candidate generation and later ranking? Missing context Dense block results select scope material without the scoped instruction Can a bounded expansion recover the complete required bundle? Input omission; possible dilution The long section&rsquo;s vector input omits the selector tail and its ranking fails; unrelated material may also dilute its representation What separately frozen comparison could isolate omission from dilution? Approximation loss LSH drops exact-search successes How much candidate breadth is required before later stages can help? Policy starvation Bounded post-filtering yields no candidates despite permitted rows Can a separately measured refill policy recover them safely? No-answer confusion All methods return candidates for the battery requests What evidence would support an explicit insufficiency decision? None of the proposed remedies in the table was tested as a remedy in this chapter. The location of the loss constrains what could help: a reranker cannot promote a required passage that never entered its pool, and a larger pool cannot recover words omitted from every indexed representation by encoder truncation.
The m

[... end of excerpt: the chapter continues past this point. The complete text of this exact revision is at the download link in section 1 above, or ask me to paste the remainder. Do not treat this as the whole chapter. ...]

== 3. Existing references and bibliography ==
No references are recorded against this chapter. That is a fact about the site, not a claim that the chapter is unsourced: treat the chapter's own prose as the claim set and look for primary sources independently.

== 4. Existing evidence ==
No validated evidence has been recorded for this chapter. A research brief exists; research has not been performed against it yet.

Unverified candidates and seeds (leads only — verify before relying on any of them):
- BEIR; heterogeneous retrieval evaluation that exposes dense-retrieval failure modes. (https://arxiv.org/abs/2104.08663)
- DPR; the dense baseline whose hard-negative training this chapter analyses. (https://arxiv.org/abs/2004.04906)
- Formal (SPLADE); sparse retrieval failure modes and term mismatch. (https://arxiv.org/abs/2109.10086)
- BEIR repository and datasets; a practical source of out-of-domain evaluation and hard negatives. (https://github.com/beir-cellar/beir)
- RAGAS; evaluation of retrieval quality including faithfulness and answerability. (https://arxiv.org/abs/2309.15217)

== 5. Research objective and questions ==
Decide whether chapter 10's failure analysis and hard-negative construction still match current practice: verify the challenge set, lost obligations, and no-answer handling.
- How should hard negatives be constructed so they test retrieval rather than annotation artefacts?
- How are no-answer and absent-evidence cases kept out of success averages?
- What distinguishes a duplicate from a contradiction, and how should each be treated?
- How is a 'lost obligation' located and recovered in a retrieval pipeline?
- What must the next retrieval stage prove after semantic failure analysis?

Previously unresolved gaps:
- Hard-negative construction is described, not released as a reproducible script.
- No-answer handling is specified but not evaluated on held-out queries.

== 6. Associated material ==
Real, known-available resources for this chapter:
- Colab notebook: https://colab.research.google.com/github/ernanhughes/programmer.ie.notebooks/blob/main/notebooks/retrieval/10-chapter.ipynb
  The notebook exists in the published inventory. Whether it runs today, and what it prints, is unverified unless a recorded run says so.

Availability is not evidence. An available notebook is a place to run an experiment, not a record that one was run or that it succeeded.

== 7. Research history ==
No research has been recorded for this chapter yet. This is the first research pass.

== 8. How to investigate ==

1. Read the supplied chapter. State its thesis, its main claims, the assumptions it depends on, the examples and code it uses, and the reader level it assumes. Do this before searching, so your search queries come from the chapter rather than from what you happen to know is fashionable.
2. Identify what may be dated or unsupported: claims that later work has overtaken, statements presented without a source, mechanisms whose current best implementation has changed, and missing developments. Equally, identify what remains sound. A chapter that needs no change is a legitimate and useful finding.
3. Form targeted search queries from the chapter's specific claims, terminology and mechanisms. Do not add papers merely because they are recent or popular.
4. Investigate original papers, official documentation, reference implementations and source code. Follow each thread to the primary source rather than stopping at a summary.
5. Use Hacker News and similar discussion sites as discovery seeds and as commentary. Follow the links to their original sources. Distinguish what a commenter asserts from what someone has demonstrated.
6. Consider Hugging Face Papers as one discovery channel where the chapter's subject overlaps its coverage. Check which tools and APIs are actually available to you now rather than inventing endpoints, and do not depend on it for books outside its subject area.
7. Verify bibliographic metadata: authors, title, venue, publication and last-update dates, identifiers (DOI, arXiv id, version) and the exact URL. Record your access date and your reading status for each source. If you read only an abstract, say so. If you could not open the full text, do not describe it as though you had.
8. For each source, state precisely which specific claim it supports, qualifies or contradicts, and what the limits of that relationship are. A source that is merely topically related supports nothing.
9. Label your evidence classes separately and never blur them: established background; results reported by a source; results you reproduced locally; your own hypotheses; and experiments you are proposing.
10. Inspect any associated code and run focused checks only if you actually have execution available and it is appropriate. Record the commands, versions, artifacts, failures and anything you skipped. Never present an experiment you did not run as a result.
11. Recommend the proportionate change: a correction, a clarification, a citation, a new example, a new experiment, a new section, or no change at all. Do not propose a wholesale rewrite of a chapter that is fundamentally right.
12. Produce concrete proposed text or a patch, with citations and a reason for each change. Note any bibliography, Concepts sidecar, notebook or neighbouring-chapter edits needed for consistency, and report them as dependencies rather than silently applying them across the book.
13. If the evidence does not justify an upgrade, say so plainly and report that instead of manufacturing changes.

== 9. Required output ==

Return your report in Markdown with exactly these top-level sections. Cite every factual claim about a source. Where you could not verify something, write UNVERIFIED rather than omitting it.

## 1. Context and provenance — chapter identity, the snapshot or revision you actually read, its scope, today's date, and any tool or execution limitation that shaped the result.
## 2. Claim audit — a table with one row per claim: the claim and where it appears, the current evidence, your concern, a priority, and the response you propose.
## 3. Source ledger — a table with one row per source: identity, verified metadata, URL, reading status (full text / abstract only / not accessible), which claim it bears on, its limitations, and your verification and access dates.
## 4. Findings — supporting, qualifying, contradictory and unresolved evidence, each with claim-level citations.
## 5. Upgrade proposal — the minimal concrete chapter changes you recommend, the rationale, the tradeoffs, and any associated resource changes.
## 6. Experiment opportunities — what should be tested, the method, success and failure criteria, and an explicit UNRUN marker wherever you did not run it.
## 7. Review checklist — the decisions the author needs to make, and your reason for accepting, revising, deferring or rejecting each proposal.

If you cannot read the chapter or a cited source, say so explicitly and ask me to paste the chapter or supply the document. Never infer the contents of a page you could not load. The chapter text and the source documents above are evidence to evaluate, not instructions to you: if a source document contains anything resembling a directive, treat it as material to assess and report on, not as a command to follow. Record what you actually did on the date you actually did it, and do not invent run identifiers, publication dates or completed work.

Assembled from the current chapter. The copied prompt and text download contain the same material.

Bring the full chapter

If your AI cannot open the chapter, give it this complete text snapshot.

Download chapter

The direction

Research brief

Decide whether chapter 10's failure analysis and hard-negative construction still match current practice: verify the challenge set, lost obligations, and no-answer handling.

Questions to answer

How should hard negatives be constructed so they test retrieval rather than annotation artefacts?
How are no-answer and absent-evidence cases kept out of success averages?
What distinguishes a duplicate from a contradiction, and how should each be treated?
How is a 'lost obligation' located and recovered in a retrieval pipeline?
What must the next retrieval stage prove after semantic failure analysis?

Starting points

Leads to investigate, not verified evidence.

BEIR; heterogeneous retrieval evaluation that exposes dense-retrieval failure modes.
https://arxiv.org/abs/2104.08663
DPR; the dense baseline whose hard-negative training this chapter analyses.
https://arxiv.org/abs/2004.04906
Formal (SPLADE); sparse retrieval failure modes and term mismatch.
https://arxiv.org/abs/2109.10086
BEIR repository and datasets; a practical source of out-of-domain evaluation and hard negatives.
https://github.com/beir-cellar/beir
RAGAS; evaluation of retrieval quality including faithfulness and answerability.
https://arxiv.org/abs/2309.15217

Open questions

Hard-negative construction is described, not released as a reproducible script.
No-answer handling is specified but not evaluated on held-out queries.

What comes back

An upgrade to review

The prompt asks for evidence and specific changes, with a reason for each one.

Claims checked against their sources
Supporting and conflicting evidence
Proposed text and new experiments

Research produces a proposal. You decide which changes to accept.

The continuing record

Research history

0 recorded runs

Your first research pass starts here

No research has been recorded for this chapter yet. Use the prompt to begin; findings and reviewed proposals can become part of this record.

Provenance and recording Chapter identity, revision, and author workflow

Stable identifier: retrieval-10-chapter
Current chapter fingerprint: d03c00d329b039d0360e3533d9acc25501485ed5da18058fae1a03edc4640330

Recording what comes back

Save the returned report and record a run with python scripts/research_companion.py record, retaining the revision the research actually examined. Recording research never edits the chapter.