Retrieval From First Principles
How a system discovers, ranks, filters, and selects evidence from a larger information world, and how to establish whether that evidence improved the outcome.
A search engine can tell us what matched. It cannot tell us what should be trusted, what should be excluded, or whether the choice helped.
Retrieval is the missing link between the much larger information world and the small set of evidence an AI system is allowed to use. It is not embedding, memory, context assembly, generation, or evaluation; those are neighbouring boundaries. Retrieval owns discovery, ranking, filtering, fusion, reranking, and selection of candidate evidence under policy and budget constraints.
Why this book exists
A large corpus can contain the right passage, yet a request can still fail: the term the user used did not match the document vocabulary, the retriever returned a near-duplicate, the passage was topically close but operationally useless, a restriction was applied after the wrong stage, or a persuasive answer was generated from evidence that should never have been admitted. The question is not only whether search found something plausible, but whether it found the right evidence, excluded the wrong evidence, and changed the outcome.
The central distinction
The book keeps these stages separate:
- Available information — everything the system could potentially inspect.
- Retrieved evidence — what the search process found.
- Eligible evidence — what the request’s scope and policy permit.
- Selected evidence — what the retrieval budget allows through.
- Admitted context — what the model actually receives.
- Used evidence — what we can establish the output relied on.
- Helpful evidence — what improves independently checked task behaviour under a declared comparison.
A high similarity score is not admissibility. A plausible answer is not proof that the retrieved evidence was used. Each transition has to be justified separately.
The cumulative experiment
The book builds one retrieval system progressively. It starts with a deterministic lexical search over a tiny frozen corpus, then adds identities, chunking, semantic retrieval, filters, fusion, reranking, evidence budgets, complex retrieval controllers, evaluation, and finally a local consumer application with read-only retrieval tools.
Its constructed collections include exact identifiers and jargon, paraphrases, multi-fact questions, contradictions and stale revisions, access-restricted material, and declared absent-evidence requests. External collections add independently prepared judgements and task references, with their sampling and model-exposure limits retained. Measurements are linked to frozen corpora, query splits, configurations and artefacts.
Chapter plan
The thirty-chapter manuscript spans six parts:
| Part | Chapters | Focus |
|---|---|---|
| I. Finding Information | 01-05 | Lexical search, document identity, inverted index, BM25, first baseline |
| II. Semantic Retrieval | 06-10 | Parsing and chunking, dense retrieval, ANN and index choice, filters, failure analysis |
| III. Combining Evidence | 11-15 | Candidate pipeline, hybrid search, learned sparse and multi-vector, rerankers, budgets |
| IV. Complex Retrieval | 16-21 | Routing, rewriting, decomposition, hierarchy, graph retrieval, bounded adaptive stopping |
| V. Evaluating Retrieval | 22-26 | Judgments, metrics, experiments, operations, downstream task effects |
| VI. Complete System | 27-30 | Reference contract, defensive boundary, real workflow, final proof |
Chapters are generated individually or in explicitly approved connected batches. Drafts receive independent technical and editorial review before the author decides readiness.
Build the capstone
The practical trail ends in What We Can Now Prove with a local Pi retrieval extension. Six executable milestones connect the lexical baseline, pinned dense representations, candidate fusion, parent expansion, bounded control and source-qualified tool delivery. Checkpoints in The First Search Baseline, Semantic Failure Analysis and Hard Negatives, Diversity, Coverage, and Evidence Budgets, Graph and Structured Retrieval and Latency, Cost, and Operational Reliability explain what to run and inspect along the way.
The extension searches two named local collections: the thirty chapters and an original teaching fixture. It exposes search, exact-reference opening and observable retrieval traces. Its installed Pi loader and real tools have executed; a model-driven Pi answer remains untested. These demonstrations establish working retrieval plumbing, not generated-answer correctness or a general advantage for hybrid search.
Status
All thirty chapters are present. The research enhancement has assessed the complete manuscript, with consolidated independent review still to follow. Constructed challenges retain same-author visible labels and shared task families; external judgements do not remove sampling or model-exposure uncertainty. The final local consumer ran, but all 140 task-condition outputs failed its frozen format contract. Semantic correctness, causal evidence use and helpfulness remain not measured. Writer and Improve remain not integrated. All thirty chapter flags remain draft = false by author instruction; those flags do not establish scientific or production readiness.
Related solutions
See /solutions/retrieval/ for the eventual applied home of this work.
Chapters
The Information Selection Problem
Separate information availability, candidate discovery, ranking, evidence selection, context admission, and task outcome.
Documents, Identity, and Provenance
Preserve source, document, version and passage identity through identical text, moves, edits and removal.
Building an Inverted Index
Find lexical candidates through postings while preserving exact references and the eligible collection view.
BM25: Ranking with Terms
Derive an inspectable BM25 variant from term counts, rarity and passage length, then test its limits.
The First Search Baseline
Freeze queries and passage judgements, compare lexical methods, and retain failures without confusing ranking with usefulness.
Parsing, Chunking, and Retrieval Units
Change retrieval boundaries while preserving the source evidence needed to judge them.
Dense Retrieval without Re-teaching Embeddings
Integrate a pinned sentence encoder through the existing evidence contract and inspect what similarity misses.
Exact Search, ANN, and Index Choice
Measure approximation against an exact reference without confusing neighbour agreement with evidence relevance.
Filters, Scope, and Freshness
Gate stale-index candidates against a trusted current view and expose the cost of filtering after a bounded search.
Semantic Failure Analysis and Hard Negatives
Use controlled misleading neighbours to locate failures before proposing another retrieval stage.
Candidate Generation as a Pipeline
Build a traceable pool of eligible evidence and measure what later ranking can no longer recover.
Hybrid Search and Reciprocal Rank Fusion
Combine method rankings precisely, then test whether agreement improves the final evidence order.
Learned Sparse and Multi-Vector Retrieval
Inspect learned vocabulary weighting and late interaction without mistaking hand calculations for trained retrieval results.
Rerankers and Cross-Encoders
Measure a pinned local cross-encoder while preserving the candidate ceiling, input limits and actual cost.
Diversity, Coverage, and Evidence Budgets
Select whole evidence units under an explicit content budget and distinguish relevance, redundancy and sufficiency.
Understanding Queries and Routing
Choose a bounded retrieval route while preserving the original request and separating intent guesses from evidence outcomes.
Query Rewriting and Expansion
Expose every query transformation and measure gains, uncertainty and extra search work without changing the information need.
Decomposition and Multi-Hop Retrieval
Retrieve a supported chain across documents while preserving bridge identity, source evidence and stopping conditions.
Hierarchical and Parent-Child Retrieval
Navigate exact document structure, restore parent context and compare granularity under a declared content cap.
Graph and Structured Retrieval
Traverse typed, source-backed relationships without treating mentions, names or paths as proof of an answer.
Bounded Adaptive Retrieval: When to Stop
Decide from observed evidence whether to stop or run one more search within a fixed budget, and distinguish a justified continuation from a successful outcome.
Relevance Judgments and Test Collections
Define information needs, preserve judgement provenance and keep unknown relevance distinct from absent evidence.
Metrics that Answer Different Questions
Calculate ranking, coverage and stopping measures without confusing a better score with sufficient evidence or a correct answer.
Controlled Experiments and Regression Gates
Freeze a retrieval comparison, isolate observation-conditioned stopping and retain query-level gains, losses and resource differences.
Latency, Cost, and Operational Reliability
Measure retrieval work at explicit boundaries and compare quality, latency and resource use without inventing one universal winner.
Does Better Retrieval Improve the Task?
Execute a local consumer, trace actual evidence admission and distinguish answer-format failure from verified task correctness.
The Reference Retrieval Contract
Turn source identity, permitted scope, selection and admission limits into explicit, testable promises to a consumer.
Defending the Retrieval Boundary
Keep untrusted source content separate from caller authority, and test exact reference, permission, admission and output boundaries.
Improve and Writer: A Real Retrieval Workflow
Compose the reference retriever with a local consumer while retaining application control over admission, review and side effects.
What We Can Now Prove
Reconstruct the executable system, retain its negative results and state the precise limits of the book's evidence.