# Candidate Generation as a Pipeline
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/11-chapter/
# Prose fingerprint (sha256, normalized): dc65e380a948eb9e6962a454552510a1d946d21a1b3e6062493b1461a66923f3
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

The maintenance request requires two pieces of R17 evidence: the statement limiting where the procedure applies and the instruction itself. In the retained block-level development trace, BM25 ranks the scope sentence first, while exact dense search ranks the code block first. Neither first result satisfies the complete authored evidence bundle. Taken together in one candidate pool, they do.
Semantic Failure Analysis and Hard Negatives showed that a seemingly relevant result can leave part of an evidence obligation unmet. Here we combine the two already frozen first-result lists by exact reference. Neither individual list covers the required bundle; the two-item union does. That is a change in what evidence reaches the next stage, not proof that either individual passage is sufficient. It is also a deterministic reuse of a fictional development trace, not an independent validation or another model run.
The engineering task is therefore not merely to retrieve more candidates. It is to give later stages a sufficiently complete, eligible pool while preserving exactly which methods contributed each candidate and what the pool still lacks.
Give every generator a common output contract BM25 and dense similarity can both return evidence, but their scores have different meanings and scales. A common candidate interface should preserve each method’s original score and rank, not force those numbers into a misleading shared measure. Combining the lists and choosing an order are separate operations.
Our combination layer wraps the existing lexical and exact dense indexes. Each candidate retains its complete source, document, version, representation, segmentation and passage reference, together with its parent-content digest and exact source span. For every generator that found it, the record also retains the method, original score, one-based rank, collection view and representation or model provenance. One passage can therefore have two retrieval origins without losing either.
The stages remain distinct. First we produce method-specific lists; then we form a pool; only afterwards do fusion, optional reranking and bounded selection determine what may be exposed:
eligible manifest units -> independent method lists -> exact-reference union -> fusion -> optional reranking -> bounded selection -> current-view projection Here “independent” means that the lexical and dense candidate lists were produced by separate methods. It does not mean their errors are statistically independent. As Part II showed, both methods can prefer the same plausible but insufficient neighbour.
The diagram labels the same stages with the question each one leaves for the next. Every later chapter in this part extends one of those boundaries; none of them changes an earlier one.
%%{init: {"flowchart": {"wrappingWidth": 470}}}%% flowchart TD U["Eligible manifest units"] --> M["Independent method lists: BM25 and dense"] M --> P["Exact-reference union — the candidate pool, and the ceiling for every later selector"] P --> F["Fusion and optional reranking"] F --> S["Bounded selection: count or character budget"] S --> C["Current-view projection — a stale reference can still be refused here"] Two boundaries matter more than the boxes between them. The pool fixes an upper bound that no later ordering can exceed, and the final projection can still refuse a candidate that earlier stages accepted.
In examples/evidence_pipeline.py, the interface between those methods and the shared pool is deliberately small:
lists = { "bm25": method_candidates(lexical_hits, "bm25", view, lexical_provenance), "dense": method_candidates(dense_hits, "dense", view, model_fingerprint), } pool = candidate_union(lists) The wrapper checks finite scores, unique references and eligibility in the declared view. The union refuses candidates with conflicting text or spans under the same reference, and it refuses lists from different views. Its output has a deterministic reference order, but that order is only a stable way to inspect the pool. It is not a relevance ranking.
Combine obligations with OR before demanding agreement Taking the intersection would keep only passages found by both methods. That can be a deliberate precision-oriented choice, but it is the wrong operation when the methods find complementary parts of an answer. The opening BM25 and dense first results have different exact references: their intersection is empty, even though their union contains both required pieces.
Union keeps every candidate supplied by either generator, deduplicating only when the complete evidence reference matches. Two sources publishing identical words remain separate records with separate attribution. When both methods retrieve the same passage, it appears once in the pool with two method-origin records. A later selector may consider redundant content, but candidate union must not discard source identity in the process.
Generator depth controls how many candidates each method offers; the final evidence budget controls what survives selection. Two ten-item lists may share references and yield fewer than twenty distinct candidates. Even when their union includes the required evidence, fusion can rank it too low or a content budget can exclude it. Pool membership tells us what a later stage could select, not what it ultimately will select.
Change the evidence, not just the wording Part II’s protocol-held-out questions changed the wording of development needs but reused their evidence judgements. That was useful for diagnosing mechanisms, not for testing against independently prepared relevance labels. We retain the fictional collection for implementation examples and add SciFact as a separate final evaluation collection with external document-level annotations.
Following the original SciFact release and annotation contract, we use all 5,183 corpus documents, each represented by its title and abstract. A declared hash rule selects twenty claims from SciFact’s public development split without consulting retrieval scores or positive labels. Twelve sampled claims have annotated positive documents; eight have no judged positives. This is the final sample for this book experiment, not SciFact’s hidden official test set.
SciFact’s original evidence-document annotations provide positive document judgements made outside this book, covering evidence that supports or contradicts a claim. They do not identify the exact passage spans our earlier bundle protocol requires, verify the authority or truth of the source, or establish that an answer would be correct. The labels are public and the twenty-claim sample is small. We can describe what happened under these judgements, but not claim population-wide performance from them.
These document judgements are external to our implementation author, but that independence has a separate limit. The dense model’s documented training sources include S2ORC data, so some SciFact documents may have been encountered during training. We have not audited exposure for the sampled documents or claims, or measured any resulting effect on retrieval. Possible training overlap is not proof of exposure to a particular item, nor evidence of a bias in a known direction. We retain both facts: the judgements were externally prepared, and model exposure remains unresolved.
The fictional development collection continues to use complete source-anchor bundles as its success criterion. SciFact instead asks how many externally judged-positive documents reach the candidate pool. These metrics answer different questions, with different units and denominators; we report them separately rather than treating one as confirmation of the other.
Measure the ceiling before judging the ranking For a claim with judged-positive documents, candidate recall is the fraction of those documents found anywhere in the candidate pool. That fraction is an upper bound on the document recall any later selector or reranker restricted to this pool can achieve. The final top-three selection may achieve less: changing the order can promote evidence already present, but it cannot create a missing candidate.
The frozen external results show:
Depth per generator BM25 pool recall Dense pool recall Union pool recall Mean union size 1 .8333 .5833 .8333 1.55 3 .8333 .6833 .8500 4.95 10 .8333 .8667 .8667 16.70 The recall columns are macro averages over the twelve claims with annotated positives; mean pool size covers all twenty sampled claims. At depths three and ten, union provides a higher candidate-recall ceiling than BM25 alone. At depth ten, however, its .8667 recall equals dense search’s .8667: union adds candidates without recovering any additional judged-positive documents on average. Increasing depth has diminishing measured returns in this sample, not a guaranteed limit for other collections.
The recorded ceiling and its cost. Recall is the fraction of judged-positive documents present anywhere in the pool, averaged over the twelve claims that have annotated positives; pool size averages all twenty sampled claims. Union adds candidates at every depth while recovering no further positives between depth three and depth ten. Claim 873 shows what those aggregate values conceal. Five documents are annotated positive. BM25 retrieves none of them within ten candidates; dense search retrieves two, so the union has document recall .4 for that claim. For claim 1368, the annotated positive document is absent even from the depth-ten union. In that second case, reranking the existing pool cannot retrieve the missing evidence, however accurately it orders what remains.
The BM25 baseline already ranks a judged-positive document first for ten of the twelve claims with positives. Only claims 1368 and 873 can improve their first-relevant rank over that baseline. That leaves little room for gains in a first-relevant-rank metric on this sample, but it does not cap the possible improvement in deeper document coverage, Recall@k or nDCG: a query can place one positive first while still missing others. We should read any later fusion result against this limited headroom, not as a broad verdict on combining retrievers.
The fictional development collection shows a different kind of gain. Across its nine answerable needs, the union covers 7/9 complete evidence bundles at generator depth one, 8/9 at depth three and 9/9 at depth ten. Those are source-bundle results on authored fixtures, not the SciFact document-recall measure. Adding their numerators or averaging their percentages together would erase the distinction this book is trying to preserve.
Preserve the boundary when the view changes The generators begin with eligible manifest units, but eligibility can change before a selected packet is exposed. Final projection therefore checks the trusted current view again, along with each exact reference, parent digest and source span. A stale or removed reference can be rejected even if it was present in the earlier candidate pool. Public output contains only permitted packets and their permitted character count; if none survive, it returns a generic empty result. The fuller offline audit is not the public response.
These checks establish behaviour under the repository’s caller-supplied, trusted views. They do not authenticate a caller, discover the latest production policy or establish a deployed access-control boundary. Retrieved text remains data: no instruction inside a passage is authorised to alter the selected view, invoke tools or widen access.
We can now identify which generators supplied a candidate, preserve its provenance and measure the evidence ceiling before ranking. We can also identify what is missing from the pool and therefore impossible for later ordering to recover. The next question is how to rank the candidates that did arrive. Hybrid Search and Reciprocal Rank Fusion combines lists by their positions without pretending that BM25 scores and dense similarities share a numerical scale.
References and further reading Background and extensions for this chapter:
Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP. Provides a dense first-stage candidate generator for a larger question-answering pipeline. Nogueira and Cho (2019). Passage Re-ranking with BERT. arXiv preprint, v1. Places learned reranking after initial passage retrieval. Lin et al. (2021). Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations. arXiv preprint, v1. Provides a reproducible implementation example spanning sparse, dense and hybrid candidate generation. Related chapters Bring Back Only What You Need. Continues from retrieved candidates to context-admission decisions.