# Does Better Retrieval Improve the Task?
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/26-chapter/
# Prose fingerprint (sha256, normalized): dbd7362b210319f42ed0e6c231976a8c295a3e2a2fb766be95c311a90824a566
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

The question asks which university was associated with the literary discussion group that Adam Fox joined. Lexical retrieval finds the relevant sources, and both annotated supporting passages enter the model’s actual prompt. The evidence has crossed the retrieval and admission boundaries. Yet the consumer describes evidence labels and document names instead of giving the required answer with citations. The failure occurs after the evidence reaches the model.
The evidence was retrieved, selected and admitted, but the generated response did not satisfy the application’s output contract. Those are separate outcomes, and this chapter measures them separately.
Latency, Cost, and Operational Reliability measured the cost of finding and selecting evidence. The next boundary is more demanding: once evidence enters a consumer’s prompt, does it contribute to a valid task output? A retrieval score cannot answer that question; we must observe the actual prompt and response.
Execute a consumer before claiming a task outcome To cross that boundary, we run an already cached small TinyLlama chat model with its actual tokenizer and supplied chat template. Execution is local on the recorded GPU, with fixed weights, greedy decoding, no tools and no external provider. Each request supplies a fixed system instruction, the original question and labelled evidence data. This controls the composition we test; it does not establish that the consumer is suitable for every task or output format.
The model generates its own response. It is not given a reference answer or a supporting-fact list from which to copy a scripted result. The ordinary application cannot see the gold labels; a separate grading step examines its output afterward.
Research on retrieval-augmented generation motivates pairing retrieval with a generator, but an architecture described in that literature is not automatically the system we executed. This experiment combines the fixed retrieval mechanisms developed in the book with one fixed local chat consumer; it neither reproduces the cited trained architecture nor imports its reported performance.
In the inspected RAG manuscript (§2.4), the query encoder and generator are fine-tuned for the task while the document encoder and index stay fixed. Our retriever and chat consumer are not trained together for this composition. The comparison matters because supplying relevant context to a fixed chat model tests an interface between components, not the effectiveness of jointly learned retriever–generator behaviour.
Use a fresh task slice and retain its limits The external questions, reference answers and sentence-level supporting facts come from HotpotQA, not from the book’s fictional fixtures. A declared hash ordering selects four development and twelve final questions from its public validation data. Eleven final questions require a bridge and one is a comparison question. Settings are fixed, so there is no separate policy-selection slice. Public evaluation material supplies independent annotations, not a blind or previously unseen benchmark.
The searchable collection is not all of Wikipedia. It is the union of the sixteen selected tasks’ supplied context sets: 160 distinct paragraph observations. Those upstream fullwiki contexts were themselves obtained through retrieval, so an annotated supporting paragraph can be missing before any of our own retrievers runs. This fixes a small, inspectable candidate universe but limits what its failures can mean.
The availability audit finds complete annotated supporting facts for only four final tasks. Thirteen final supporting-fact annotations have no corresponding source reference in the bounded collection. These are availability losses upstream of our ranking and admission stages. We preserve the missing references instead of fetching them after seeing results or treating a partial set of available gold evidence as a complete upper bound.
A missing annotated supporting paragraph does not prove that no other passage could help answer the question: supporting-fact annotations are not exhaustive relevance judgements. For contrast, a separate fictional inventory declares genuine missing-evidence, single-fact, list and injection controls against a fully specified local collection. Those same-author controls probe mechanisms; they do not turn into independent external results.
The HotpotQA questions and labels are public, and we have not audited exposure during model training. Development and final tasks have no overlapping question identities or known-positive document identities, but the restricted collection retains the upstream retrieval choices. The split therefore supports a frozen local comparison without establishing model-training independence or deployment generalisation.
Follow the actual input, not the selected list We trace the entire evidence path: available source references, retrieved candidates, eligibility, selection and admission. The assembler rechecks selected references against the trusted view provided with the request, builds the actual chat messages and measures them with the consumer’s tokenizer. This verifies what entered the prompt under that supplied view. It does not discover a more recent authorisation state or establish production authentication.
The admission contract permits at most two whole evidence units and 900 tokens for the evidence JSON, within a 1,536-token complete prompt and a separate allowance of 64 generation tokens. Labels, source references, JSON wrappers and chat-template text consume part of those limits. We therefore measure the assembled messages with the real tokenizer; character lengths and separately tokenised fields cannot be treated as an exact substitute for the final prompt length.
When a selected unit cannot fit, the assembler records a capacity exclusion rather than clipping that unit and pretending the shortened text is the same evidence. On final questions, lexical retrieval selects four units on average but admits two; dense also selects four on average but admits about 1.83. Identical maximum allowances do not yield identical admitted packets. This is why selection metrics cannot stand in for evidence delivered to the consumer.
The gap is observable at the query level. For one lexical question, selection contains all annotated support but admission retains only part of it; dense has two such questions. A report that stopped at the selected list would count those cases as evidence successes even though the complete annotated support never entered the prompt.
Compare fixed conditions Every external question is run under seven fixed conditions: no retrieval; lexical, dense and hybrid retrieval; the unchanged bounded adaptive controller; and two available-support diagnostics. The diagnostics place up to one or two annotated supporting paragraphs into the prompt in fixed source order. They use gold labels only for this explicitly labelled diagnostic purpose, never to drive an ordinary retrieval policy.
The consumer, instruction, decoding settings, required output format and scorer are identical across conditions. Ordinary retrieval keeps its previously declared call, candidate and selection ceilings. The assembler applies the same maximum admission and token budgets to every condition, while recording the different evidence packets that actually fit. Equal ceilings therefore control the allowance, not the realised input.
Condition Retrieval calls across twelve tasks All annotated support admitted No retrieval 0 0 Lexical 12 2 Dense 12 1 Hybrid 24 1 Adaptive 21 1 Available-support, two units 0 diagnostic calls 4 Available-support, one unit 0 diagnostic calls 0 The table measures whether the annotated supporting spans reached the prompt, not whether the model answered correctly. The diagnostic rows are not faster retrieval alternatives: their zero retrieval-call count reflects direct access to external annotations that an ordinary search procedure cannot use. In particular, four complete-support diagnostic packets do not establish four correct answers.
The mean fraction of annotated supporting facts actually admitted is approximately .417 for lexical retrieval and .347 for dense, hybrid and adaptive retrieval. These values differ from support fractions calculated at selection. They locate losses along the evidence pipeline, but the annotation set is incomplete and the fractions do not constitute exhaustive document relevance or downstream answer-quality measures.
Report the consumer failure without rescuing the score The frozen application contract requires two output lines: an answer line followed by admitted evidence labels or an explicit no-citation marker. Not one of the 84 final generations follows it. The same failure occurs in all twenty-eight development generations and twenty-eight constructed-control generations. This is a direct observation of the consumer’s interface, not an inference from retrieval scores.
Under the declared parser and scorer, every condition receives zero format-constrained exact match and token F1. That result is valid for this application contract: the outputs could not be parsed as required answers with citations. It does not determine whether their unrestricted prose is factually correct, whether retrieval changed their content, or whether one retrieval condition was semantically more helpful than another. Uniform zeros after a common interface failure cannot support those comparisons.
The neutral feasibility probe showed that the model loaded and could generate meaningful text; it did not test the stricter answer-and-citation interface. All twenty-eight development outputs were complete, and their invalid format was observable before final generation. The protocol had not declared a consumer-format or answer-capability gate that would require stopping, so continuing was an experimental-design weakness rather than violation of a mandatory gate. We report that weakness and keep the frozen prompt, parser, model and questions unchanged instead of engineering a successful format after seeing the final outcomes.
The retained generations show why a permissive parser tweak is not a defensible rescue. In a case-insensitive marker search, 133 outputs contain neither required marker and seven contain an answer-like marker without a citation marker. None contains CITES: in any case, and none begins with the specified ANSWER: form. Allowing different capitalization or marker positions would therefore recover no complete contract-compliant output. The parser enforces the specification that was frozen before evaluation.
Sixty-six outputs end with an end-of-sequence token and remain invalid; the other seventy-four reach the generation limit. Because invalid outputs also terminate normally, truncation alone cannot explain the universal format failure. Instruction placement, long evidence identifiers, lack of demonstrations and model capability are possible contributors, but their individual effects were not measured. Evidence position was not varied. Lost in the Middle studies position effects for the models and tasks in that paper, not the cause of this consumer’s failure. A better prompt or model might behave differently, but this experiment does not measure that counterfactual. Crucially, 140 malformed records are not 140 independently verified factual errors.
The retained counts distinguish executable observations from questions the failed output interface prevented us from evaluating. In particular, the zero format-constrained scores must be read alongside the two explicitly not measured outcomes at the bottom of the table.
What the run recorded Value Task-condition generations 140: 28 development, 84 final, 28 constructed Outputs satisfying the frozen answer-and-citation format 0 Outputs containing a citation marker 0 Outputs ending with an end-of-sequence token 66, none of them format-valid Outputs reaching the generation limit 74 Format-constrained exact match and token F1, every condition 0 Semantic answer correctness not measured Evidence use by the consumer not measured A new, separately authorised consumer experiment would need development-stage checks for format compliance and basic answer capability, a predeclared rule for stopping before final evaluation if those checks fail, and a frozen consumer configuration. It would also need fresh final questions and independent assessment of answer correctness and citation support. These are requirements for a future study, not gates retroactively imposed on the experiment already completed.
eRAG offers one possible direction: its §2 runs a consumer on an individual document and uses the downstream task score to form a document-level label. Such a label is conditional on that consumer, prompt and scorer. Because our consumer failed its required output format, its format-gated zero scores are not interpretable as measures of a document’s semantic usefulness and cannot be recycled as eRAG-style relevance labels.
Single-document usefulness also misses dependencies between pieces of evidence. As a hypothetical example, document A names a bridge entity and document B supplies the entity’s attribute. A consumer might fail with either document alone and succeed with the pair. Labelling both documents useless from their isolated zero scores would lose that interaction. This is a conceptual counterexample to single-document evaluation, not a measured eRAG result or an observed success of our TinyLlama consumer. Joint evidence needs their own declared set-level test.
Inspect answers without inventing independent judgement The deterministic evaluator uses predeclared normalised exact match and token-overlap rules, checked against the primary HotpotQA evaluator. HotpotQA provides independently prepared reference answers, but those references do not supply an independent semantic assessment of every malformed response. Recalculating the fixed format-gated score is not the same as judging the free prose.
The opening example illustrates the observable mismatch: instead of returning the university, the raw response discusses the Inklings and Adam Fox evidence labels. Another response says an actress had no other jobs although the external reference gives modelling. We can inspect these disagreements against the recorded text and references. We cannot convert selected examples into an independently assessed semantic accuracy rate.
The fictional launch-code control exposes a different risk: one response repeats the declared code while also echoing hostile instructions from retrieved source text. The missing-procedure controls express a lack of instructions in ordinary prose instead of the specified formal abstention. Because no output satisfies the required format, parsed abstention and citation rates cannot be established. Zero valid records do not establish that raw outputs never abstained or never made unsupported assertions.
The assembled prompt proves which source text was delivered. A compliant citation could be checked against admitted references, but the required citation structure was absent. Whether an answer is semantically supported, whether the consumer causally relied on a passage, and whether a person found the answer helpful remain further questions. Neither an echoed fact nor a malformed answer settles them.
What this experiment answers This chapter establishes three outcomes: a real local consumer executed, retrieval policies differed in which annotated evidence reached its prompt, and all generated outputs failed a frozen answer-and-citation contract. It does not establish whether better retrieval improves semantic answer correctness, whether a shorter evidence packet works just as well, or whether the controller’s retrieval losses become downstream answer losses. A common failure at the output interface prevents that inference rather than resolving it.
The next engineering obligation is not another unmeasured ranking improvement. It is a reference interface that makes source identity, permitted evidence, admission and failure states dependable for a consumer. The Reference Retrieval Contract specifies those promises and how to test them, without mistaking successful retrieval for a successful answer.
References and further reading Background and extensions for this chapter:
Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. Provides a concrete retriever–generator composition whose task outcome can be measured. Liu et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL. Measures evidence-position effects in its tested QA and key-value tasks; it does not test this chapter’s consumer or explain its format failure. Salemi and Zamani (2024). Evaluating Retrieval Quality in Retrieval-Augmented Generation. arXiv preprint, v1. Derives per-document labels from a particular consumer’s scored task outputs; those labels require an interpretable score for the intended outcome. Related chapters Did It Reach the Model?. Checks whether assigned evidence actually reaches the model. Did the Context Help?. Separates delivery from measured behavioural benefit.