The Information Selection Problem

An engineer asks how to restore a device to its factory settings. The fictional handbook contains the instruction, “R17 reset: hold the blue button for ten seconds to erase all setup.” Yet a search for “restore factory settings” returns nothing. The answer-bearing passage is present; the search rule cannot connect the engineer’s words to the handbook’s.

The engineer tries “R17 reset” instead. This time the search finds both the device handbook and a network handbook. Their scores tie, so an identity-based tie-break puts the device instruction first. The order is reproducible, but the query has not established which operation the engineer intended. A deterministic result can still be the wrong result.

One query misses available evidence; another finds competing interpretations and arbitrarily selects one. Before introducing an index or a learned representation, we will reproduce both failures with a search rule small enough to calculate by hand. The aim is to identify which decision was made at which stage—and what that decision cannot establish.

An information need and a query

Established background. An information need is what a person or task is trying to establish; a query is the expression submitted to a retrieval system. Those are not the same thing. Manning, Raghavan and Schütze make this distinction in Introduction to Information Retrieval, Chapter 1, §1.1, printed p. 5. Matching the submitted words is one possible mechanism for satisfying the need, not the definition of success.

Our first scan takes a frozen corpus, a query string and a positive result-count limit k. It records every positive match in a deterministic order, selects the first k passages and returns an inspectable trace—not an answer. That is enough to examine where candidate discovery ends and selection begins; the fuller evidence contract will follow from the failures.

A corpus small enough to inspect

The repository file examples/fixtures/toy-v1.json contains the following original fictional passages. These are teaching data, not instructions for real equipment.

In this fixture, each passage is its entire document. Its span is a half-open range of character positions in the original text: the start is included and the end excluded. Positions count Python string code points, not UTF-8 bytes. Case folding and tokenisation may change the searchable representation, but they do not change the retained source text or its span.

Consider device-reset: its source is toy:device-handbook, its document is device-r17, its version is v1, and its locator is toy:device-handbook/device-r17/v1#reset. That locator is an internal fixture reference, not a live URL. These fields tell us which recorded passage the search returned. A similarity score alone cannot identify its source or recover the original words.

These fields are only a starting point. They do not yet handle document moves, revisions or citations into parsed files; Chapter 02 will make those identities precise. Even here, identical words from distinct sources must remain distinct evidence records.

The smallest ranking mechanism

Our rule is a design choice for this example. Convert the query and passage into sets of case-folded word tokens. Score a passage by the number of distinct query tokens also present in it:

score(query, passage) = size(query_terms intersect passage_terms)

Discard zero scores and order the remaining candidates by descending score, breaking ties by source, document, version and passage identity. Then select the first k, with k = 1 in this example. The tie-break is a reproducibility rule, not an interpretation of user intent.

This budget counts passages, not their length, tokens or cost. Repeated query words do not increase the score because we use sets. There is no stemming, synonym expansion, stop list, phrase matching or learned model.

For “R17 blue button”, the query set is {r17, blue, button}. The device passage matches all three terms; the network passage matches only r17; the export passage matches none. Their scores are therefore three, one and zero. The export passage remains available but fails candidate discovery under our positive-score rule. Both handbook passages are retrieved; the one-passage selection limit retains only the device passage.

This calculation explains an order, not a probability of correctness. Dividing each score by the fixed query length would rescale the overlap score without calibrating its confidence.

The later regression controls make the representation’s limits concrete. On the original corpus, R17 matches while R-17 does not. AB-12 and AB 12 produce identical term sets, as do C++ and C#; the common word the retrieves both handbook passages. These are recorded properties of this tokeniser, not estimates of how frequently such failures occur or evidence that a stop list would improve results.

Run the scan

Implementation status: implemented and executed. examples/keyword_search.py uses only Python’s standard library. The following is its scoring loop, with the query terms and candidate list initialised as in the implementation:

query_terms = terms(query)
candidates = []
for passage in corpus["passages"]:
    matched = sorted(query_terms & terms(passage["text"]))
    if matched:
        candidates.append({**passage, "score": len(matched), "matched": matched})

Here terms is set(re.findall(r"\w+", text.casefold())). The complete file also validates the fixture, applies the tie rule and returns the trace. It copies the original passage fields into each candidate rather than reconstructing a citation from the normalised terms.

Run these commands from the repository root:

python examples/keyword_search.py "R17 blue button"
python -m unittest discover -s tests -v

The command prints JSON, not an answer. Compare retrieved with selected, and inspect the matched terms on the selected record: they explain the three-point score and the one-passage cutoff. status reports the operation’s outcome, not whether the user’s need was satisfied. The loader accepts the public synthetic fixture and rejects inconsistent whole-document spans; direct calls to the underlying search() function do not enforce source scope.

There is no index yet. Every query scans the passage texts and constructs their term sets again. Its work grows with the amount of text scanned, and retaining all matches also requires sorting and storing those candidates. We have not measured latency or memory. An inverted index will address a specific source of repeated work later; it is unnecessary to understand the contract first.

Hold the corpus still and change the query

Book result: Chapter 01 keyword scan. The frozen run applies the same scan to five queries and records candidate scores, ordering and the selected passage. The inventory, tokenisation, public scope, tie-break and k = 1 stay fixed; only the query changes. The unchanged inventory is an availability control, not a query-dependent retrieval method. Information needs and relevance judgements in examples/fixtures/ch01-dev-queries.json were supplied by the author, not inferred or independently adjudicated by the scanner. The hand calculation above illustrates the rule; the table below reports the retained execution.

The observed ordering and scores are stored in experiments/ch01-keyword/frozen/observations.json, with configuration, input/output digests, environment and test transcript alongside it. The result and its limits are recorded in research/evidence-register.md.

The exact query retrieves and selects its intended passage. The paraphrase “restore factory settings” retrieves nothing because none of its terms appears in any passage under this tokeniser. The evidence is available, but lost at candidate discovery, before ranking or selection could help. An improved tie-break would not repair that miss.

The ambiguous query is different: discovery finds both reset passages with equal scores. The identity tie-break puts the device first, and top-one selection hides the network alternative from a consumer that sees only the selected prefix. We have established an ordering rule, not which interpretation the engineer wanted. Clarification would be appropriate for the authored need, but this scanner neither detects ambiguity nor asks a follow-up question.

The two no-answer requests expose the opposite mistake. Q99 calibration returns no candidates, consistent with the absence of supporting evidence in this fixture. But the factory-settings paraphrase also returned none even though supporting evidence existed: no_candidates cannot mean “the corpus has no answer”. Conversely, the battery-replacement query matches r17 and returns a passage, although neither handbook supplies a replacement procedure. A non-empty candidate set does not prove answerability.

This is a diagnostic comparison, not a contest between ranked retrieval systems. All five queries and their authored judgements are visible development examples. There is no held-out or independently adjudicated evaluation, no measured quality improvement and no basis for extrapolating these observations to a real collection.

To replay the recorded comparison without overwriting it, run:

python experiments/ch01-keyword/run.py --output .artifacts/ch01-replay

Compare the replay’s observations.json with the frozen file. Environment details and test durations may differ; the query traces should agree for the recorded inputs. The experiment README documents the protocol and the retained failed validation attempt.

The original eight-test transcript is a historical record. Later revision tests are retained in a separate package that replays the original query traces exactly and names its test modules explicitly. Because the original runner used broad test discovery, its current test count differs from the historical transcript; that change does not add evidence to the original result.

A match can still be outside scope

Matching well is not permission to use a passage. Suppose a record shares more query terms than the public handbook but is outside the request’s allowed scope. We call a passage eligible when the applicable trusted scope and policy permit its use. The separate fictional examples/eligibility_demo.py entry point demonstrates that boundary by checking scope before passing any text to the unchanged scanner:

if record.get("scope") != PUBLIC_SCOPE:
    continue

Only records explicitly labelled public reach the scanner; missing or unknown scope is excluded. The public response counts and describes only that permitted view, without exposing excluded text, identifiers, scores, locators or rejection details. Run the bounded fictional demonstration:

python -B examples/eligibility_demo.py "R17 reset blue"

Book result: Chapter 01 revision checks. In the separate frozen revision capture, this entry point selects the permitted passage. Changing or adding excluded records leaves its public response unchanged. Tests check that excluded text never reaches the scorer and that outputs and diagnostics omit the excluded markers. Direct use of the underlying scanner selects the excluded synthetic passage in the negative control: bypass is possible, not prevented. The evidence is in experiments/ch01-revision/frozen/, separately registered in research/evidence-register.md.

Book result: Chapter 01 final correction diagnostic. A separate offline inspection of the retained fictional scanner and fixture records the following scores for R17 reset blue:

The unprotected scanner ranks the excluded synthetic passage first because it shares more query terms. The public wrapper excludes it before scoring, allowing the permitted passage to be selected. The excluded score in the table comes from offline inspection; the wrapper never exposes it. In this prefiltered view, eligible mirrors the retrieved passage IDs because exclusion happened earlier. The all-public original fixture also has matching eligible and retrieved IDs, but that alone demonstrates no exclusion. This later diagnostic is retained separately in experiments/ch01-r2/frozen/ and the evidence register; the earlier revision evidence is unchanged.

This is a trusted-label teaching example, not authentication or a general access-control system. The scope metadata is assumed to be correct; the raw scanner remains callable, and neither a storage boundary nor timing-side-channel protection is established. Chapter 09 returns to scope and freshness in more detail.

A permitted source can still be wrong, stale or contradictory. The scanner does not reason about negation or authority. Retrieved prose is data: returning a passage does not execute instructions inside it or grant those instructions authority over a later consumer’s tools and policy.

The contract these failures motivate

We can now name three different losses. The paraphrase fails at discovery. The reset alternative survives ranking but is hidden at selection. An out-of-scope match fails eligibility before the scan sees it. To reason about what happens after selection, the book uses this evidence chain:

available information -> retrieved -> eligible -> selected -> admitted -> used -> helpful

The figure maps those three losses onto the chain and marks which stages the chapter actually observed. Read it from the top: only the first four rows describe measured behaviour here.

This is a logical accounting model, not a mandatory execution sequence. In the scope example, the eligibility decision occurs before retrieval. A system handling protected sources must enforce its access boundary before exposing candidates, snippets or traces; the diagram does not authorise reading or returning excluded material.

These boundaries also define the book’s scope. Retrieval discovers, filters, ranks and selects evidence candidates. Embedding supplies representations; memory retains information over time; context assembly decides what enters a model invocation; generation produces the response; and evaluation checks behaviour. No successful retrieval stage can stand in for proof that the next stage succeeded.

A search score cannot close the chain

Imagine that a downstream consumer receives the device passage for the ambiguous “R17 reset” request and repeats its instruction. This is hypothetical, not an observed model output. We would still need to establish which operation the engineer intended, what text was admitted, whether the consumer relied on it and whether the resulting action or answer was appropriate. Nothing in a deterministic search score supplies those checks.

Established background. The information retrieval literature distinguishes result relevance from broader user utility, including effects of presentation and summaries. See Introduction to Information Retrieval, Chapter 8.

The cited discussion motivates distinguishing ranked relevance from user utility; it does not demonstrate that this scanner improves an AI task. Admission, reliance and helpfulness require their own observations and comparisons later in the book.

The original focused tests retain their checks on matching, provenance, tie stability and fixture validation. Separate revision tests document tokenisation behaviour and the bounded scope example, including its bypass limitation. The new runner records both test modules, complete input snapshots and hashes, commands and outputs. To replay it, choose a fresh output directory:

python -B experiments/ch01-revision/run.py --output .artifacts/ch01-r1-replay

These checks reproduce the declared scanner and wrapper behaviour. They do not validate a general permission system or show that a generated answer improved. Admission, use and helpfulness remain not measured.

What we can now inspect

We now have an executable first retriever and a trace for locating failure. On a corpus small enough to inspect, we can identify a lexical miss despite available evidence, explain a score term by term, see a tie hidden by top-one selection and check a trusted public-view exclusion. Each observation answers a narrower question than “did the system help?” The Introduction: Retrieval From First Principles gives the wider evidence chain that the next chapters will test.

The next chapter, Documents, Identity, and Provenance, makes that inspection survive changes to the collection. Its question is precise: when a source is renamed, revised or duplicated, how do we preserve the identity of the evidence we actually retrieved?

References and further reading

Background and extensions for this chapter:

• Broder (2002). A taxonomy of web search. ACM SIGIR Forum. Connects a submitted query to several possible user intents.
• Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. Extends the selection problem into a retriever–generator system.
• Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks. Motivates testing selection across different tasks and domains.

Related chapters

• Retrieval Is Not Geometry. Distinguishes retrieval success from properties of the vector space.