Bring Back Only What You Need

Chapter 13 left the model holding a tidy live context and a shelf of external artifacts:

Incident 17 — lock-order inversion.
Evidence: artifact://incident-17

plus five more references of the same shape. Then the current task changes. The user asks why the deadlock appeared only under concurrent import. The incident artifact probably matters. So might the benchmark report, the migration plan, and the compiler run. The naive response is to load everything, which recreates within one turn the exact occupancy problem Chapter 13 solved. Leaving was only half the problem. Existence plus recoverability does not mean admission. This chapter decides what comes back.

Retrieved is not admitted

The chapter’s central equation, protected throughout:

retrieved
   ≠
admitted
   ≠
used
   ≠
helpful

Four different claims. An artifact surfaced by some search is a candidate. A candidate inserted into the bundle is admitted. Admitted material the model actually references is used. And helpful means the task improved because of it, which no earlier stage guarantees. The pipeline this chapter owns is the middle of that chain:

EXTERNAL INFORMATION
        ↓
candidate generation
        ↓
CANDIDATES
        ↓
admission
     ↙   ↓   ↘
 reject admit expand
        ↓
CONTEXT BUNDLE
        ↓
MODEL

Everything above the candidate set is some other mechanism’s business: exact identifier lookup, file lookup, text search, BM25, embedding retrieval, database query, graph lookup, tool search, agent navigation, memory recall. The book does not teach these, compare them, or rank them. They all end at the same boundary, a candidate set, and Chapter 14 begins there. Everything below admission is behaviour, measured downstream. The chapter lives in the admission box: which candidates enter, in what representation, at what cost, and whether the entering material helped.

Candidates are not context

Candidate generation identifies information that may be relevant to a computation. Admission determines which representation of those candidates is actually allowed into the context bundle.

The distinction matters because each stage fails differently. Imagine ten external artifacts, three containing information the current task requires. The generator returns seven: the three required plus four irrelevant. Candidate generation achieved perfect recall. Admitting all seven may still fail the computation through bloat, interference, position pressure, and cost. Retrieval recall is not admission precision, and a mediocre generator feeding a strong admission policy can outperform a strong generator feeding no policy at all. The primary experiment therefore freezes the candidate set deliberately: construct candidate pools directly, hold generation constant, and vary only admission. If the chapter changed the retriever and the gate at once, the result would be uninterpretable.

A later pipeline can fail at either stage, and the vocabulary keeps them apart. A candidate miss means the required artifact never entered the candidate set; later retrieval work owns that. An admission miss means the required artifact sat in the candidates and was rejected or never expanded; context engineering owns that. The frozen-candidate experiment studies the second failure with the first held fixed, and an optional ecological extension runs one simple real generator afterwards to check the admission policy still behaves under imperfect candidates. That extension is subordinate. The chapter does not depend on it.

The cost of admitting everything

Condition B of the experiment, preload everything, is the baseline every other policy must beat, and the reasons it can lose are now measurable rather than atmospheric. Candidate-level precision understates the damage. One relevant 10-token artifact plus one irrelevant 20,000-token artifact is 50 per cent candidate precision and near-total token waste. The chapter’s most important metric is therefore token-level: irrelevant admitted tokens counted alongside required admitted tokens wherever oracle labels make the split meaningful.

That irrelevant tokens merely waste money would be a budgeting complaint. The evidence says they can actively harm. Amiraz and colleagues formalise the distracting effect of an irrelevant passage with respect to a query and a model, measured as the probability that the model fails to abstain when the passage cannot answer the question, and show the scores correlate strongly across seven tested models from 3 to 70 billion parameters. Their headline behavioural result is the one this chapter imports: on public question-answering benchmarks, adding a hard distracting passage alongside the gold passage drops answer accuracy by 6 to 11 points depending on the model, and the degradation persists at 70-billion scale. Two of their secondary findings sharpen the point for admission design. Higher-ranked irrelevant passages are more likely to distract, and adding a reranking stage amplifies the effect: the passages that fool the pipeline are also the passages that mislead the generator. Fixed top-k admission over a strong ranker is therefore not a neutral default. It preferentially admits exactly the material most likely to distract. Their population is question answering over retrieved passages, not agentic artifact admission, and the chapter claims nothing beyond the narrow finding: retrieved-but-irrelevant can degrade generation, andCuconasu and colleagues’ earlier split stands — random passages do not degrade, distracting ones do.

Over-admission can thus resurrect everything Chapter 5 buried: bloat, interference, and position pressure without ever touching the hard context limit. Externalisation solved residency. Admission decides whether the solution survives contact with the next task.

Relevance is not utility

Similarity scoring ends at the candidate boundary, and the chapter states the boundary without re-teaching embedding geometry. A document can be highly similar to the task and still be redundant, outdated, lower-authority, too broad, or simply not required; a dissimilar artifact can hold the one exact constraint the task needs. Retrieval score is not context utility. Top-ranked is not authoritative either: a project note is not a user instruction, and Chapter 14 preserves provenance and authority metadata through admission without adjudicating conflicts. Chapter 19 owns precedence. This chapter keeps the metadata Chapter 19 will need, including which source each admitted section came from, so no chunk arrives as anonymous prose.

Utility is also marginal, not per-document. Two independently correct documents can repeat each other, consume capacity, shift positions, and increase reasoning burden. Consider a migration task with three retrieved candidates: the migration plan stating tenant 042 needs manual cutover, a progress note restating the same cutover requirement, and the compiler run showing the current build state. The first document carries the decision. The second adds no new fact the task needs; admitting it anyway spends tokens, shifts the plan’s position, and doubles the model’s opportunity to misread the tenant identifier. The experiment tests one sufficient source against three redundant ones without building a major new axis, to establish the working definition:

Admission utility is task-relative: the behavioural value of adding a representation to the current bundle, net of the context resources and interference it consumes.

No scalar score is constructed from this. Task success, tokens, latency, cost, and interference stay separate dimensions for the eventual compiler to trade off. Admission is marginal context utility, not document relevance alone.

How much of a candidate should enter

A candidate artifact is not atomic. A 30,000-token investigation may owe the current task one decision, one exact identifier, or one evidence section, and the admission choices range across identifier only, title plus metadata, semantic anchor, specific section, selected evidence span, and full artifact. That range is admission granularity, and it is not Chapter 12 fidelity wearing new clothes:

FIDELITY
How detailed is this representation?

GRANULARITY
How much of the candidate does this representation cover?

A full-fidelity hundred-line section and a low-fidelity whole-document summary differ on both axes at once. The experiment manipulates one at a time: a granularity arm compares full artifact against relevant section against short typed extraction under matched informational requirements, asking whether admitting the whole candidate imposes unnecessary cost relative to the needed part. Where the external artifact already carries Chapter 12’s tiered representations, admission consumes them, choosing which level enters for this task without rebuilding the fidelity policy. And Chapter 7’s retention semantics ride along: a candidate holding exact constraint material admits it exactly, because no admission pressure compresses anything below its legal fidelity.

Admission also composes with what is already resident, and the composed states deserve names because the experiment will produce them constantly. A resident anchor plus an admitted section is the normal working shape: eighty tokens of orientation that never left, joined by the two hundred lines the current question actually needs. A resident anchor plus an admitted full artifact is the escalation shape, reached only through EXPAND after the section proved insufficient. Two admitted sections from the same artifact are one source admitted twice, and the token ledger should show it that way rather than as two independent wins. The admission record keeps these compositions visible so evaluation can distinguish a policy that leans on resident anchors well from one that re-admits what the bundle already knew.

Progressive disclosure

The natural strategy these pieces compose into is progressive disclosure: identity first, then anchor, then section, then full artifact, each step admitted only when the current material proves insufficient.

IDENTITY
artifact name / title / metadata
        ↓
ANCHOR
short semantic description
        ↓
SECTION
targeted material
        ↓
FULL
complete artifact

Chapter 12 pre-generated multiple resident representations and selected among them over time. Chapter 14 progressively admits more information from a non-resident candidate. The direction of travel is opposite — fading out versus drawing in — and the chapter keeps the terms apart for exactly that reason.

Disclosure is not free, and the chapter prices both sides. Lean starts buy lower initial occupancy and less irrelevant admission; they pay additional tool calls, latency, extra reasoning steps, and two characteristic failures. The first is premature stopping: the model sees enough to believe it understands and never loads the necessary detail. The fixture builds this deliberately, a candidate whose anchor reads sufficient while its detail section carries the decisive qualification — an incident anchor recording the fix with a detail section confining it to the import path while the export path still deadlocks. Required expansion available but not requested is scored as an admission miss inside progressive disclosure, plainly named and unbranded. The second failure is wandering: expanding A, B, and C, searching D, reading E, after the correct artifact was already identified, at the cost of latency, calls, and context. Anthropic’s context-engineering guidance names this risk directly, warning that agents navigating on demand can chase dead ends or miss key information without good heuristics. The chapter takes that as engineering evidence for the failure mode, not as law about its frequency.

Whether the model or the runtime drives disclosure is an architecture choice the chapter compares without crowning. In system-selected admission a deterministic gate chooses candidates and expansions; in agent-selected admission the model chooses what to inspect next. The experiment runs one of each, which also sets up the handoff Chapter 15 owns: an agent that retrieves one misleading candidate may form the hypothesis that sends it after supporting material, a self-reinforcing loop this chapter names and leaves alone.

Eager admission and just-in-time admission inherit the same even-handed treatment. Eager preloading buys low round-trip latency and immediate evidence at the price of over-admission; just-in-time starts lean at the price of steps, misses, and wandering. The trade is sharpest where the task’s information needs are predictable: a debugging task that always needs the failing test output plus the suspect module rewards eager admission of both, while an exploratory task with six plausible artifacts rewards starting from identities. Anthropic’s guidance describes production hybrids, stable or critical material preloaded with the rest explored on demand, and the chapter derives the same shape from its own costs before citing it: preload what every near-future computation needs, disclose the rest. Final assembly policy still belongs to Chapter 22.

Fixed top-k gets one short section because it is really a policy wearing a parameter’s clothes. Retrieving five and inserting five couples ranking and admission into a single number with no theory behind it. The right count varies with artifact size, task complexity, candidate redundancy, evidence diversity, context pressure, and retrieval confidence, and k knows none of these. The chapter builds no dynamic optimiser. It establishes the coupling and moves on.

The admission machinery, minimally

Two conceptual records carry the mechanism into the laboratory, specified here and implemented nowhere. A candidate record holds candidate identity, source artifact identity, the generating mechanism, its rank or score where one exists — some generators supply none, and score = unavailable is a legal value — plus representation, token cost, provenance, and scope. An admission decision holds the candidate, one of ADMIT, REJECT, DEFER, or EXPAND, the selected representation, a reason, token cost, and policy version. DEFER means not enough evidence to spend tokens yet; EXPAND means admit a richer representation or section. No workflow semantics beyond those two words.

Three identity rules keep the machinery honest. Stable artifact identities from Chapter 13 survive the whole pipeline, candidate generation through expansion to evaluation, so results attribute to artifacts rather than to fuzzy phrases. Admitted sections keep their source: artifact:incident-17 with section: "Lock acquisition sequence", never anonymous prose. Fragment identity is not artifact identity: five retrieved chunks may come from one artifact, and candidate counts that hide that are misleading. Chunking strategy itself is not taught; the relationship is preserved. Duplicate or overlapping candidates are labelled explicitly in fixtures, and candidate diversity is observed as an admission consideration, with semantic deduplication declared out of scope.

Recovery integrity comes before admission quality, inherited whole from Chapter 13. Did the reference resolve to the intended artifact and version is asked before was this the right candidate to admit. A failed lookup is a recovery failure, never a ranking failure, and the decomposition survives into the capstone: candidate-generation records flow into admission decisions which flow into the bundle, so evaluation can always say which stage failed.

Admission’s neighbours each get one line. Ordering among admitted candidates uses a fixed deterministic order in the primary experiment, with ordering effects declared a confound if found; Chapter 6 and the Chapter 22 assembly own the general problem. Newly admitted material records admission position, first divergence, and mutation radius for Chapter 9’s cache ledger, but correct information comes first and cache optimisation is never the admission objective. Freshness is controlled by version-pinned frozen artifacts; Chapter 20 owns the live problem. Scope is controlled the same way; Chapter 21 owns leakage. Memory candidates and tool outputs enter as candidates like any others; Chapters 16 and 17 own their generation.

Proposed experiments

Deterministic fixtures carry the chapter, with hidden ground truth withheld from every admission policy. The external set holds twelve to twenty artifacts across several types. For one target task: two required, two helpful but unnecessary, four plausible distractors, the remainder irrelevant. Ground-truth categories are fixture labels, never model-visible: REQUIRED holds information necessary for correct completion; HELPFUL may aid reasoning but is not necessary; DISTRACTOR is plausible and topically related but unnecessary; IRRELEVANT is unrelated; TRAP looks highly relevant but contains misleading or conflicting detail, constructed so the fixture knows the truth. Traps stress admission, not authority: Chapter 19 owns competing authoritative claims, so traps mislead by emphasis and omission rather than by asking the model to adjudicate sources.

Three fixtures deserve names. One large distractor, highly plausible and not required, exposes the gap between candidate precision and token waste without a full retrieval benchmark. One small decisive candidate, low surface similarity and required, tests whether largest or most similar is being confused with most useful; candidate ranking stays frozen, so this is not an embedding experiment. One disclosure trap, sufficient-looking anchor over detail that reverses the conclusion, tests premature stopping.

Experiment 1: admission over a frozen candidate set. Fixed tasks, fixed artifacts, fixed candidate pool, fixed ranking, fixed model, fixed resident baseline; only the admission policy varies. A, no external admission, testing whether external information is needed at all. B, preload all candidates, high recall with terrible precision. C, fixed top-k at a pre-registered k, never tuned after results. D, an admission gate over the same pool, generation unchanged. E, identifier-first progressive disclosure with targeted expansion, extra steps measured. F, oracle admission of exactly the minimum sufficient set, where the oracle may admit part of an artifact rather than the whole, and where the oracle’s forbidden knowledge is stated in the run record. The granularity arm follows for required artifacts: full against section against short typed extraction, one axis at a time.

Measurement separates the stages the chapter separated in prose. Candidate recall and precision for the generator; admission recall and admission precision for the gate, each defined against the oracle set only where fixtures make requiredness checkable — open-ended real work offers no such labels, and the chapter does not pretend otherwise. Token-level accounting underneath: total, required, irrelevant, and distractor admitted tokens, reported per task so a policy cannot hide one expensive mistake inside a favourable average. The ledger also counts what the admission machinery itself consumes: candidate summaries read, ranking tokens burned, expansion tool calls made. A gate that saves 8,000 context tokens by spending 6,000 reasoning about candidates has a net to report, not a victory to declare. Context management does not get to hide its own context usage. Occupancy around the decision: pre-admission bundle, admitted size, post-admission size, usable-budget utilisation, connecting back to Chapter 4 without re-teaching it. Downstream behaviour: task success, critical evidence use, unsupported claims, constraint adherence, plus latency and call counts where useful, always preserving available-but-not-admitted versus admitted-but-unused as different failures. The second of those deserves a concrete picture: the compiler-run section sits in the bundle with the exact flag the task needs, the model’s answer never touches it, and the run fails for want of what was already present. That is not a retrieval problem and not an admission problem; it is a use problem, and a chapter that merged it with the others would prescribe gate tuning for a reasoning failure. Bounded pre-registered remove/restore ablations on a subset of admitted candidates test helpfulness directly: run with the candidate, run without it, everything else constant. No exhaustive leave-one-out; the point is materiality, not completeness. Against the minimal oracle set, policies read as under-admitting or over-admitting, kept only where sufficiency is checkable.

Falsification is stated before any run. Preload-all performing as well or better under realistic budgets, gating that saves few tokens while adding real latency, disclosure that chronically stops early, agent exploration that wastes more than eager loading, ranking quality dominating so admission contributes nothing, simple top-k matching elaborate gates, irrelevant candidates causing no behavioural harm in the tested models, minimum sufficient sets so large that policy offers no savings: each shrinks the mechanism. Negative results simplify the book on purpose. If top-k matches the gate, the gate is deleted — but the metrics stay, because candidate precision, irrelevant admitted tokens, and the miss decomposition are what let the next experiment see clearly. If preload-all wins under real budgets, that is said plainly, and the chapter’s surviving contribution is the accounting that proved it rather than the gate that lost. If identifier-first disclosure costs unacceptable latency, elegance does not save it. The capstone gets simpler when evidence permits.

Context Lab and corpus implications

The laboratory gains two concepts and the ordering between them. Candidate-generation records precede admission decisions, each admission pointing at its candidate, each candidate pointing at its source artifact, generator, and score-or-unavailable. Fields enter by experiment consumption only: candidate and source identities, generator and rank, representation, token cost, provenance, scope, decision among admit, reject, defer, and expand, selected representation, reason, admitted tokens, policy version. Nothing is implemented in this task; Project Context stays in corpus-collection mode until Chapter 14 freezes these requirements, and a later implementation stage may add them then.

The corpus position is unchanged and stated without embarrassment. Project Context holds zero genuine ecological sessions at drafting time, so the chapter’s causal experiment is synthetic by design. Later traces can answer prevalence questions fixtures cannot: how often external artifacts are referenced, how often agents re-read large files, how many candidates ordinary work surfaces, how much context arrives through search and read tools. They cannot answer which artifact was truly necessary. Controlled fixtures remain necessary for admission quality, and the chapter never asks the corpus for what it cannot give.

What the agent creates next

Admission assumed a universe of candidates the agent moves through. The next chapter removes that comfort: agents do not merely retrieve context, they manufacture it. Plans, hypotheses, tool observations, summaries, and intermediate artifacts continuously create new future candidates, including the misleading ones whose retrieval can steer everything after. Retrieval generates possibilities and admission creates context; what happens when the agent starts generating the possibilities themselves is the problem Chapter 15 inherits.

References

• Anthropic Applied AI team (Rajasekaran, Dixon, Ryan, Hadfield, et al.). “Effective context engineering for AI agents.” First-party engineering essay, September 2025, verified September 2026. Embedding and pre-inference retrieval; just-in-time context with lightweight identifiers; agentic navigation with progressive disclosure; hybrid preload plus runtime exploration; dead-end exploration risk. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• Jones, A., Kelly, C. “Code execution with MCP: Building more efficient agents.” First-party engineering essay, Anthropic, November 2025, verified September 2026. Progressive disclosure via filesystem-presented tools and search_tools with detail-level selection; filtering data before it enters context; keeping large intermediate results out of the model context. https://www.anthropic.com/engineering/code-execution-with-mcp
• Amiraz, C., Cuconasu, F., Filice, S., Karnin, Z. “The Distracting Effect: Understanding Irrelevant Passages in RAG.” Preprint, arXiv:2505.06914, May 2025 (arXiv page lists a related 2025 ACL proceedings DOI). Distracting-effect measure with cross-model robustness; hard distracting passages cut accuracy 6–11 points even alongside the gold passage; higher-ranked irrelevant passages distract more. https://arxiv.org/abs/2505.06914
• boxpositron. “WithContext MCP Server.” Third-party implementation, MIT licence at review, v3.0.6, verified September 2026. Search/read workflow with project-scoped notes; explicit write-out and read-back separation. https://github.com/boxpositron/with-context-mcp