# The First Search Baseline
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/05-chapter/
# Prose fingerprint (sha256, normalized): 9cc1393003d584b89e35b29aaff2b888baae3fc7980611e2b81cc4b6b3083906
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

The command passage now ranks first for cache reset. That is the result we designed the Chapter 04 example to illustrate. It shows how BM25 can change an ordering; it does not yet show that BM25 is generally the better ranking method.
To evaluate that claim, we need information needs and relevance judgements fixed before inspecting the comparison. We also need to hold the searchable collection and candidate policy steady while changing the ranking mechanism. Otherwise, we can keep choosing examples that favour whichever method we have just built.
BM25: Ranking with Terms gave us an explicit scoring rule. Here we turn Part I into a reproducible comparison: the original scan, indexed overlap and BM25 search the same eligible passages, and their results are checked against declared relevance judgements. The question changes from can we explain this score? to what difference did this scoring rule actually make?
Separate the questions before choosing a metric An index can reproduce the scan exactly and still retrieve unhelpful evidence. A ranker can improve judged result positions while increasing computational cost. A passage can be relevant to the need yet forbidden by policy. We should not ask one metric to answer all three questions.
Question Check in this baseline Retrieval correctness Scan/index agreement on identities and overlap scores; posting and reference tests Ranking quality Judged passage Recall at fixed cutoffs and truncated reciprocal rank Query latency not measured Memory/storage cost Canonical index-JSON bytes recorded; resident memory not measured Source preservation Exact reference reconstruction and duplicate-source survival Eligibility Per-view filtering, lifecycle checks and restricted-content invariance Answerability Authored corpus judgement, distinct from presence of lexical candidates Downstream task usefulness not measured These checks answer different questions. Better judged ranking cannot certify source truth, permission or a correct downstream answer; lower execution cost cannot compensate for searching the wrong evidence. We will report each claim at the boundary where we can test it.
Freeze the collection and information needs The baseline view in examples/fixtures/part1-v1.json extends our fictional collection without replacing its earlier records. It retains the distinct specification and blog copies of the reset instruction, then adds focused and verbose cache notes, network instructions, a rare key, accented labels and a negative battery statement. All retain Chapter 02’s source, document, version and passage identities.
The permitted view contains thirteen passages. Every method receives those same eligible passages. The scan adapter calls the original keyword_search.search unchanged; indexed overlap obtains candidates through postings but retains the scan’s distinct-term score; BM25 ranks those same candidates using its fixed parameters. Restricted material contributes neither candidates nor collection statistics. We are comparing ranking rules, not changing the evidence each method is permitted to inspect.
The query files are separate:
examples/fixtures/part1-development.json examples/fixtures/part1-heldout.json Each file declares eight information needs, their query strings, slice labels and a complete set of binary relevance judgements for the eligible passages. Development queries cover exact matches, focused versus verbose instructions, accents, a rare identifier, repeated words, paraphrases and absent procedures. The test queries use other wordings and combinations, including repeated source occurrences. A query’s label describes its authored need, not what the search happened to return.
Relevance belongs to the information need, not merely to overlapping words. R17 battery replacement requests a procedure. A note stating that replacement is unsupported and giving no procedure fails that need, even though its vocabulary matches. The same note could be relevant to a different question—is replacement supported?—without any change to its text. The judgement has to say which question is being answered.
The lamp query has a different obligation: find all public occurrences of the check. Its four relevant passages include repeated paragraphs in two source copies. Because these are distinct evidence references, returning one does not satisfy the full authored need. A request for just one distinct instruction might call for different judgements or deduplication. We keep the original denominator instead of changing it after seeing the ranking.
What “held out” means here The first protocol seal fixes hashes of the corpus, both query files and scoring configuration before implementation. A second seal fixes the index and evaluation code, tests and runner before the first test-split evaluation. k1=1.2 and b=0.75 were chosen in advance and not tuned. Regression tests draw on development queries and separate arithmetic controls, not on expected rankings from the held-out slice.
The boundary is real but limited. One implementing agent authored the synthetic corpus, questions and judgements, and their content was visible during development. Protocol-held-out therefore means the recorded test evaluation occurred after the protocol and code were sealed; it does not mean blind assessment, independent authorship or a representative sample. Hashes make changes auditable. They cannot create independence that the experimental design lacks.
The sequence below is what the seals order. Read it as a record of when each artefact was fixed, not as a claim that the test slice is independent.
%%{init: {"flowchart": {"wrappingWidth": 460}}}%% flowchart TD A["Corpus, both query files and scoring configuration fixed<br/>hashes recorded · k1 = 1.2, b = 0.75 declared here"] --> B["Index, evaluation code, tests and runner fixed · hashes recorded"] B --> C["First evaluation on the protocol-held-out split"] C --> D["Results inspected and retained as a fixed baseline"] D --> E["Development queries for diagnosis; newly constructed cases for later claims"] D -.->|"regression tests use development queries, not expected held-out rankings"| E The dotted annotation matters as much as the solid ones: the parameters were fixed at the first seal, and no regression test encodes a held-out ranking.
Once the test results have been inspected, they are no longer unseen evidence for future tuning. We retain this run as a fixed baseline, use development material for further diagnosis and require newly constructed, preferably independently authored cases for any later claim of generalisation. Reusing the same test errors to improve a method and then reporting the old split as untouched would defeat the purpose of the seal.
The primary-source discussion of ranked evaluation motivates judging ranked prefixes. It does not validate our labels, query sample or split discipline. Those are local experimental decisions awaiting independent review.
Define the quantities exactly For a query, let R be the set of eligible passage IDs judged relevant to its information need, and let S_k be the first k IDs returned by a method. When R is non-empty, recall at k is:
Recall@k = |R intersect S_k| / |R| Recall@1 measures the fraction of all judged-relevant passages recovered in the first position; Recall@3 measures that fraction in the first three. A useful first hit need not produce recall one. If the need accepts two distinct source occurrences and the system returns only one, Recall@1 is one half. This is a coverage measure, not an assessment of whether the reader could complete the task.
Established background: recall measures the fraction of relevant items retrieved. We apply it to ranked prefixes under the passage-level judgements above.
Reciprocal rank asks how soon the first relevant passage appears. If its rank is r among the first three results, RR@3=1/r; if none appears by rank three, the value is zero. MRR@3 averages these values over the answerable queries. Unlike recall, it does not reward finding additional relevant passages after the first. NIST’s trec_eval reciprocal-rank implementation supplies the first-relevant-result background; the cutoff and averaging rule here are explicit local choices, not a claim of matching every tool default.
Queries with an empty R are reported separately. Recall has no denominator for them, so our evaluator returns an undefined value instead of treating no-answer cases as automatic successes or failures. We also record whether the retriever returned candidates anyway. Such a candidate is not proof that the information need can be satisfied, and an empty result is not proof that the corpus lacks supporting evidence.
For this small fixture, every eligible passage has an explicit binary judgement. The evaluator rejects missing labels, duplicate returned IDs and judgements from the wrong collection view; it never silently treats an unlabelled passage as irrelevant. That exhaustive labelling is possible because the corpus is tiny. It does not solve the incomplete-judgement problem in larger collections.
We do not calculate nDCG in this first baseline. With binary labels, Recall@k and MRR@3 already give two deliberately different views: how much evidence the prefix covers and how early its first relevant item appears. Neither quantity measures factual correctness or downstream generation quality. Later chapters introduce other measures when the retrieval problem requires them.
Read the recorded comparison Implementation status: implemented and executed. Book result: first lexical baseline. The shared frozen run reports:
Split Method Mean Recall@1 Mean Recall@3 MRR@3 Development Original scan 0.416667 0.833333 0.666667 Development Indexed overlap 0.416667 0.833333 0.666667 Development BM25 0.583333 0.833333 0.750000 Protocol-held-out Original scan 0.678571 0.821429 0.857143 Protocol-held-out Indexed overlap 0.678571 0.821429 0.857143 Protocol-held-out BM25 0.678571 0.821429 0.857143 The numbers are rounded from retained per-query calculations. Six development queries and seven protocol-held-out queries have at least one relevant passage and enter these averages. Two development queries and one held-out query have none; their relevance measures are undefined and recorded separately. All three methods nevertheless return candidates for each of those three absent-answer needs.
Book result: query-level interpretation of the retained baseline. The mean improvement on development data deserves a closer look. BM25 changes the first-ranked passage on just two of the eight queries, d2 and d7; only d2 changes the judged ranking measures for the better:
Development query First result: overlap → BM25 Recall@1: overlap → BM25 RR@3: overlap → BM25 d2, cache reset Verbose note → command passage 0 → 1 0.5 → 1 d7, cache cache Verbose note → archive 0 → 0 0.5 → 0.5 Every improvement in the development averages comes from d2: the very cache reset example used to introduce BM25 in Chapter 04. On d7, the judged command remains second; BM25 merely replaces one irrelevant first result with another. Recall@3 is identical on every development query. The method has demonstrated its mechanism, not independently confirmed an advantage beyond our motivating example.
The protocol-held-out slice provides a different result: BM25 changes the first-ranked identity on none of the eight queries. Seven have the same first passage under both methods, while h8 returns no candidates under either. The ordering lower in the list changes for h2 and h7, but coverage and first-relevant ranks remain unchanged. The aggregate equality is therefore consistent with the per-query traces, not a rounding artefact.
There is a useful limitation to this particular test: none of the protocol-held-out overlap rankings has a top-score tie between passages with different text. Some ties involve identical-text copies or repeated occurrences. BM25 can reverse a non-tied overlap order through frequency, rarity or length, so the absence of such ties is not a mathematical explanation for every unchanged result. It describes the opportunities present in these eight authored queries.
For this test slice, BM25 shows no improvement at rank one and no gain in the reported aggregates. That is neither evidence of a general BM25 advantage nor a general failure: the sample is small, constructed and not independently blind. The calculation in research/calculations/part1-rank-audit/ establishes these query-level comparisons from the existing frozen traces and judgements. It does not rerun retrieval, change labels or tune a scoring parameter.
Indexed overlap also gives exactly the scan’s judged ranking results. That is expected: its purpose is to change candidate access while preserving the original overlap ordering. We have not timed the methods, so faster search remains a design motivation, not a measured result of Part I.
Keep an error notebook beside the averages The aggregate numbers become more informative when we inspect the failures they conceal:
restore factory settings in development and wipe temporary storage in the held-out split return no candidates despite judged relevant evidence. The lexical vocabulary does not bridge those paraphrases. The negative battery note matches replacement/repair queries strongly, but supplies no requested procedure. More matching terms can strengthen the wrong answerability inference. The lamp query has four relevant occurrences, so a three-passage budget cannot cover all of them even with perfect ordering. That limit is selection capacity, not necessarily bad scoring. Repetition helps an archive passage outrank the judged cache command for the broad development query. Saturation reduces one source of distortion without understanding intent. One case even contradicts its predeclared slice name. Development query ZX99 repair was labelled no-answer-empty, yet its trace contains a match: repair appears in the negative battery note. The label did not determine the result. We keep the mismatch visible rather than renaming the slice or deleting the query. The relevance judgement is still coherent—there is no ZX99 repair procedure—even though the candidate list is non-empty.
These are diagnostic examples from fictional material, not estimates of failure frequency in real repositories. In Part II, a semantic method will need to face the entire fixed set—including exact identifiers, misleading matches and absent-answer requests—not just the paraphrases that make semantic retrieval look attractive.
Reproduce Part I The implementation is shared across all three chapters:
examples/document_identity.py exact references and trusted views examples/lexical_index.py rows, postings, overlap and BM25 examples/lexical_evaluation.py complete binary judging and metrics From the repository root:
python -B -m unittest discover -s tests -v python -B experiments/part1-baseline/run.py --verify experiments/part1-baseline/frozen python -B experiments/part1-baseline/run.py --output .artifacts/part1-replay Use a new output directory so the frozen evidence is never overwritten. Compare the replayed index.json and observations.json byte for byte with their retained counterparts. Paths, timestamps and test durations may vary; the deterministic index and query observations should not. A matching replay checks this recorded computation, not the independence of its judgements.
The package retains the corpus, query splits, complete labels, fixed scoring parameters, implementation and test snapshots, protocol seals, execution commands, environment details, protected inputs and output hashes. It runs the original tests and historical evidence verifiers as separate checks. Claim-level entries in research/evidence-register.md distinguish index correctness, the BM25 worked example and observed ranking quality. Prior frozen results are preserved rather than updated to fit the present manuscript.
The lexical evaluation module originally added twenty-four focused tests to forty-four existing tests. They cover score arithmetic, metrics, identity, representation and policy invariants. These historical counts describe that baseline package, not the size of the book’s later capstone suite. Passing them establishes the checked implementation behaviour; it cannot independently validate the authored relevance judgements or extend the result to a different corpus.
What Part I now supplies Part I now supplies an executable sequence: identify permitted source passages, build postings, retrieve candidates, rank them, and compare selected prefixes against fixed judgements. The experiments keep visible several distinctions that are easy to lose in a search interface: matching is not selection, text is not source identity, relevance is not permission, and a historical record is not automatically current evidence.
The seven-stage contract remains open after selection:
available information -> retrieved -> eligible -> selected -> admitted -> used -> helpful Beyond selection, the chain remains untested here: admission to a consumer’s context, actual use of evidence and downstream helpfulness are not measured. Nor have we established verified source authority, production access control, query latency, throughput or resident memory. The first baseline tells us what the lexical methods returned and why, not what an answer-generating system would do with those results.
The next chapter, Parsing, Chunking, and Retrieval Units, questions another choice we have treated as fixed: what counts as one searchable passage? A paragraph may omit a heading that limits an instruction, while a larger section may preserve context at the cost of more text. Before changing representation or ranking again, we need to know whether a retrieval unit actually contains the evidence that the task requires.
Our existing judgements identify exact passage IDs. Re-chunking the source creates new IDs, so transferring the old relevance labels would silently assume that every new fragment satisfies the same need. Chapter 06 instead must define source-level evidence anchors and check whether a new unit—or a selected combination of units—fully covers them. Partial overlap is not automatically relevance. That mapping has to be justified before comparing chunking schemes; this chapter neither performs the re-chunking nor retrospectively adjusts its held-out judgements.
Capstone checkpoint: the same mechanisms behind a local interface The optional capstone trail starts here, reusing the same scan, identity records, postings and BM25 code. From the repository root, run its first teaching milestone:
python -B -m examples.capstone.milestones 1 --summary The readable summary compares observed scan and BM25 rankings and checks exact reference opening on an explicitly fictional teaching collection. Full evidence is written to .artifacts/capstone/runtime/milestones/01.json; without --summary, the command retains its one-line JSON status output. Index preparation writes to an ignored cache, not the source documents or this chapter’s frozen baseline. The result is an inspectable retrieval step, not an answer. What We Can Now Prove eventually exposes this work through a Pi extension.
flowchart TD A[Named sources and exact units] --> B[Scan and BM25] B --> C[Optional pinned dense candidates] C --> D[Union and rank fusion] D --> E[Bounded selection and current checks] E --> F[Evidence contract] References and further reading Background and extensions for this chapter:
Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks. Places a lexical baseline within a broader cross-domain evaluation design. Voorhees (2000). Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Information Processing & Management. Examines how assessor variation affects comparisons between retrieval systems. Lin et al. (2021). Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations. arXiv preprint, v1. Provides a concrete example of packaging baselines with queries, judgements and evaluation scripts. Related chapters Examples Are Experimental Data. Explains fixtures, experimental data and splits by underlying task family.