# Did the Context Help? # Book: Context From First Principles # Chapter URL: https://programmer.ie/books/context/26-chapter/ # Prose fingerprint (sha256, normalized): 13195780cd9d0b39d1e6aa28863e2d7b3ac42695deaa5c41276e1ba472ab0149 # Snapshot generated at build time by Hugo. This is the complete rendered prose # of the chapter; it is not a transcript of the source file. Earlier chapters established bounded structural evidence for compiled bundles (Chapters 23 and 24) and, in Chapter 25, evidence of delivery to an instrumented host boundary for specific historical configurations. Neither establishes whether a delivered bundle changed a reader’s decision or improved the task outcome. This chapter asks: Given a controlled change in supplied context, what evidence is sufficient to attribute a behavioural difference to that context, and what evidence is sufficient to call the difference useful? The evidence is small, and its weaknesses are part of the findings rather than qualifications to hide. The principal study used six constructed tasks and one reader whose immutable model identity was not captured. A calibration control failed to measure its intended behaviour, an exploratory reader-transfer probe changed the reader and most bundle conditions together, and a candidate-removal series started from an outcome already at the scoring floor. These limits constrain causal attribution, not just generalisation. What kind of evidence this is Kind Question Structural Does the artifact satisfy declared construction constraints? Transport Was the identified treatment observed at the claimed runtime boundary? Behavioural Under a controlled intervention on context, did the response or outcome change? Ecological How often do these situations and effects occur in real work? These are different claims and not levels of one score. The corpus of real sessions is empty, so nothing here is ecological. The six-task ladder below did not use the live OpenCode path examined in Chapter 25. Its harness inserted frozen bundle text directly into a project-context section of the reader’s prompt; the no-context condition used the same wrapper with a marked empty section. Thus the treatment was assembled input text in that harness, not a separately reconciled runtime injection. The records support comparisons among the prompt conditions as constructed, subject to the unpinned reader identity, but they do not borrow Chapter 25’s transport qualification. Conversely, delivery verified in Chapter 25 cannot validate these prompt insertions: [Chapter 25](/books/context/25-chapter/) transport evidence ≠ transport evidence for these runs What is intervened on For a task, reader, bundle, prompt wrapper and decoding configuration, write the observed outcome as Y(task, reader, bundle, prompt, decoding). The intended contrast holds all arguments except the bundle fixed. This notation identifies the comparison sought; it does not establish that the serving reader or every runtime condition was actually constant. In particular, the moving model alias leaves reader identity uncertain during this wave. The unit of evidence is an observed call on a constructed task under a recorded bundle condition, not an estimate for models or compiler deployments in general. The bundles differ in content, item count, order, size and distractors, so the intervention is the bundle condition, not one particular fact. A bundle-level comparison does not identify which candidate caused a difference. Two ideas are kept apart. Influence asks whether changing the bundle changed what the reader produced, and utility asks whether that improved an outcome scored by something other than the model. Four outcome variables carry them: response-level influence: the parsed response text differs decision-level influence: the chosen action differs score-level utility: the deterministic task score differs harm-level utility: the fixture's harmful-action flag differs A changed explanation is not necessarily a changed action, and a model’s explanation does not reveal the mechanism that produced its answer. A correct answer without a counterfactual comparison does not show that context helped; mention, quotation or citation is weaker still. The proposed item-attribution ladder moves from an item being present, to being mentioned, to an outcome being consistent with its use, to outcome sensitivity when it is removed, and finally to an exact restoration that reverses the removal effect. Restoration strengthens the attribution case only when the task, reader, decoding, surrounding bundle and comparison procedure are controlled. Even then it identifies a response to an intervention, not a transparent account of the reader’s internal reasoning. The design and what was actually run Six constructed tasks were tested at the tightest of three budgets. Each had seven bundle conditions: no context; four simpler assembly methods (dump, top-k, weighted and hard-gated greedy); the compiler bundle; and a hidden-evidence oracle bundle. The supplied bundles were frozen outputs from Chapter 24’s construction experiment and were checked against recorded digests; no compilation occurred during inference. The fixtures were deliberately invented, reducing the prospect that a reader could know their particular facts from training, but that design does not prove the reader lacked all relevant prior knowledge. Responses used a fixed action schema and were parsed and scored by deterministic code against fixture truth. The grader was not a model, and its determinism does not establish the validity of the target field or the action labels. The denominators, reconstructed from the frozen records: Component Tasks Conditions Repeats Calls Primary ladder 6 7 first call only 42 Surgical variants (one task) 1 4 first call only 4 Diagnostic repeats 4 chosen cases as chosen 2 extra each 8 Main run 54 Transfer probe (second reader, one task) 1 7 1 7 The 46 “matched cases” are the 42 ladder calls plus the 4 surgical variants, all first calls. The 8 repeats are the two extra calls on each of four cases chosen before the run. Cases, conditions and calls are not interchangeable terms, and the transfer probe is a separate run. Provider-reported tokens for the main run total 6,580 input and 2,095 output, and cost is not priced because the reader is local. The principal reader was a mid-sized local instruct model addressed by a moving alias. Its immutable weight or model digest was not recorded, so its exact identity cannot be reconstructed and changes in the served weights within the wave cannot be excluded from the stored evidence. Temperature zero and a fixed seed were requested; neither guarantees identical responses across calls or establishes identical serving conditions. Repeats provide limited evidence about the stability of actions and scores on four selected cases, not proof of deterministic operation across the whole experiment. The results therefore describe the recorded calls under this partially identified reader configuration. The prompt wrapper was amended once, after an earlier pilot in which no output parsed. The tasks, truth, graders and parser were unchanged. In the main run every answer parsed, which says nothing about the parser’s general validity. The grader The grader is deterministic code, and rescoring all 54 recorded raw responses with the frozen parser and grading functions reproduced every stored score; the twelve records previously flagged harmful were flagged harmful again. That establishes replay consistency of the scoring implementation. It does not establish that the grading contract matched the task instructions. The calibration fixture below exposes precisely that defect: the prompt asked for an identifier in one field while the grader read another. Unparseable responses are assigned score zero and not harmful, without repair; this policy also means parse failures cannot safely be read as harmless behaviour. Parsing success and correct grading must be assessed separately. Inspect this in the Context Viewer The viewer scene for this chapter keeps the three evidence kinds visibly apart: the six-task ladder split into improved two, worsened one and unchanged three; the invalid calibration control marked as failed measurement; the confounded reader probe marked unsupported; and the transport-verified live wave at the score floor marked as a valid null on its recorded measures. Open the scene and read each verdict’s status before drawing any conclusion. What to examine: where influence was observed, where delivery verified without measured effect, and where the instrument itself failed. Influence observed on the ladder; delivery verified without measured effect in the live wave. Recorded results. Open the behavioural scene in the Context Viewer Results by task Task No context Supplied bundles (six conditions) stale cheap candidate abstains, 0.0 proceed, 1.0 in all six mixed pool abstains, 0.0 refuse, 1.0 in all six dependency trap uses the tool, 1.0 uses the tool, 1.0 in all six wrong scope uses project A, 1.0 uses project A, 1.0 in all six qualification trap holds, 1.0 releases, 0.0, harmful, in all six calibration abstains, 0.0 0.0 in all six Comparing no context with the compiler bundle, the recorded full parsed response differed on all six tasks and the chosen action differed on four. The deterministic score rose on two, fell on one and stayed unchanged on three. These are paired observations from the recorded calls, not a population success rate or a reliable estimate of average treatment effect. The one-task arithmetic surplus of improvements over declines is not an effect-size estimate. Because the reader was addressed by an unpinned alias, the observations support a cautious bundle-condition association rather than fully isolated causal attribution. Structural advantage did not become behavioural advantage Chapter 24 found fewer structural violations for the compiler than for the specified baselines on its synthetic corpus. That structural result did not automatically create a discriminatory behavioural test. For three of the six tasks, the compiler and hard-gated greedy conditions had matching recorded bundle digests, so those comparisons contain no identified bundle-content contrast; digest agreement was not separately checked against raw encoded prompt bytes. On the other three tasks—dependency trap, qualification trap and stale cheap—the digests differed, but neither chosen action nor score differed. Those tasks offered weak separation: the first already succeeded without context, the second failed under every supplied bundle, and the third succeeded under every supplied bundle. The supported result is no observed action- or score-level distinction in three differing-digest comparisons, not evidence that the compiler’s additional constraint machinery lacks behavioural value. The compiler and the oracle received digest-identical bundles on four of six tasks. On the two where they differed, the action and score did not change either. So on these fixtures with this reader, the compiler’s machinery beyond hard gating produced no observed decision-level or score-level difference, and neither did the oracle beyond the compiler. That is the finding, and it says little about tasks, budgets or readers where the bundles would differ in ways that matter: dependencies, complementary representations, conflicts, larger pools. The comparison ranks no assembler overall, because the structural and behavioural objectives are measured by different evidence. Response-level difference needs care. Of the four tasks where the compiler’s response differed from hard-gated greedy’s, one had an identical bundle. Repeated calls of the same case gave the same action and score all eight times, and on one case the wording differed across repeats. Wording varies without any change to the bundle, so response-level influence is noisy, and decision-level and score-level influence are the more informative variables here. The harmful case On the qualification trap the reader given no context holds. Its stated reason is insufficient information, so the correct hold is not evidence that it understood the qualification. Every supplied bundle led it to release, and release is the fixture’s harmful action: the truth file lists the harmful actions per task, the labels were frozen before inference, and harm here means an action the fixture designates as violating the desired state. The bundle the compiler built, which is also the oracle’s and the weighted packer’s, contains the claim that the migration completed and the qualification that tenant 042 remains pending. The reader released anyway. The four surgical variants of the compiler’s qualification-trap bundle also produced release: removing the qualification, restoring the parent bundle exactly by recorded digest, replacing that material with unrelated volume, and replacing it with a plausible falsehood. The parent bundle had already produced the harmful action and a zero score, so the intervention had no room to show a further scored decline. Likewise, restoration cannot be called a successful causal reversal when no deterioration appeared after removal. These variants do not identify the qualification as irrelevant, protective or harmful. The supported observation is narrower: this reader released under each recorded variant, and the qualification’s inclusion in the original bundle did not prevent that outcome. A new item-attribution fixture needs a baseline capable of moving in both directions, controlled replacement size and placement, an exact restoration condition, and independently checked bundle content. The control that did not work The calibration task states the identifier in its own prompt and asks the reader to output it exactly, with bundles borrowed from another task. It was meant to show interference. Its result is a score of zero under every condition, including no context. In every response the reader placed the identifier in the target field. The grader keys on the value field, which the prompt never named, so no response scored. The action varied, with abstention under no context and under dump and an echo elsewhere, and nothing here explains that. The calibration fixture is not a successful negative control. All conditions received score zero, but the reader placed the requested identifier in the target field in every response while the grader required it in value, a field the prompt did not ask the reader to use. The uniform zero therefore reflects a mismatch between the task’s stated response contract and the grader, not demonstrated failure to extract the identifier. Actions differed across conditions, but those differences cannot repair the invalid score or establish interference. This fixture is retained as evidence of an instrumentation defect. A repaired, versioned control should state the response schema unambiguously, demonstrate competent no-context performance and be validated before it is used to test distractor effects. Reader dependence A second, smaller local reader from another family was tested on the mixed-pool task at the medium budget and abstained in all seven conditions. This is not an exact reader-transfer experiment: moving from tight to medium budget changed five of the seven supplied bundles as well as changing the reader. The no-context and oracle bundles were recorded as identical across the two runs, yielding one narrower comparison. Both readers abstained without context; on the same recorded oracle bundle, the first reader chose the correct refusal and the second abstained. That is an exploratory action difference between two reader configurations for one task and one bundle, with no repeated matched runs or immutable reader identities. Reader-relative sufficiency remains a hypothesis to test by applying identical prompt bytes and frozen bundles to both pinned readers. The oracle is not a behavioural optimum The oracle contains the minimum evidence judged sufficient by the fixture specification. Its objective is hidden-evidence sufficiency and not the reader’s behaviour: oracle bundle ≠ behavioural optimum On the qualification trap it releases like the rest, which is objective mismatch and not oracle failure. The result motivates a hypothesis, that operational sufficiency may be reader-relative, and one reader cannot test it. Two live waves The live-runtime path was used in two waves of the same frozen design, and they must not be merged. The first, the wave of 24 runs that opens Chapter 25, produced no valid observation and contributes no behavioural evidence in either direction. Its transport failed on every run. The second was run after the harness gained the liveness gates, and it is a separate experiment from the six-task ladder above. Eight fixtures were each run under three conditions, no context, a matched distractor and the exact minimal evidence, for 24 planned runs, 24 executed and 24 with transport verified by the observer: absence observed for the no-context runs, and the expected block present exactly once for the distractor and oracle runs. The reader was pinned by a digest verified before the first call, each run was a fresh agent session, and the tasks were built so that hidden project state was needed to choose correctly. The context was authored from fixture truth and rendered by the frozen pipeline, so this tested whether correct context moves behaviour through the runtime and did not test the compiler’s selection. On the recorded measures the three conditions were identical. The task score was zero under every condition, including the oracle’s, the harm flag was raised on the same two tasks in every condition, every output parsed, the required content was absent from every output, and the evidence-use measure was zero. Verified delivery was not followed by a measured treatment effect: verified delivery ≠ measured treatment effect This wave supports a null only for its recorded outcome measures: scores stayed at zero, the same two harm flags appeared under every condition, all outputs parsed, and the required-content and evidence-use measures stayed at zero. It does not establish identity of the raw responses or of all actions, because those were not preserved and compared in the frozen result. More importantly, every condition sat at the score floor; the instrument could neither detect worsening below zero nor demonstrate that the oracle condition produced competent use of the injected facts. Reader decoding was not fixed, and the reader’s digest was pinned only for the configuration at the start. Two earlier harness launches failed and were excluded before valid runs. An unrelated plugin failed to load and the compiler package pin changed during the wave; neither changed the fixture-authored blocks, but both belong in provenance. A better wave needs a calibrated task and grader, preserved raw responses, controlled decoding, a contemporaneously qualified adapter, verified negative controls and session attribution, and at least one treatment comparison for which the measured outcome has room to improve or worsen. Transport-valid does not mean behaviourally diagnostic. Other people’s evidence Source Population Finding cited Does not establish Li et al., ContextBench (arXiv preprint, 2026) 1,136 issue-resolution tasks from 66 repositories, five coding agents Substantial gaps between explored and utilised context; scaffolding gave marginal retrieval gains Anything about these six tasks Gloaguen et al., AGENTS.md study (arXiv preprint, 2026) Coding agents on established benchmarks and repositories with committed context files Context files did not generally improve task success and raised cost by over 20% on average That supplied context never helps Modarressi et al., NoLiMa (ICML 2025) Long-context retrieval with minimal lexical overlap Effective use fell well below nominal length The behaviour of a short curated bundle Zhu et al., SWE Context Bench (arXiv preprint, 2026) 1,100 base and 376 related tasks Accurately selected prior experience helped, and unfiltered or incorrectly selected context gave limited or negative benefit That the compiler selects well These populations do not replicate this experiment. They show that the questions asked here arise elsewhere. Claim ledger Claim Evidence Status Bundle condition is associated with response differences Ladder, six tasks Observed as cases; wording-level attribution is weakened by same-bundle repeat variation Bundle condition is associated with decision differences Ladder, four of six; all eight repeats kept action and score Earned as cases, more stable than wording Bundle changes can improve the score Two tasks Earned as cases, no prevalence Bundle changes can worsen the score Qualification trap Earned as a case The compiler outperforms hard-gated greedy behaviourally None Not earned The oracle improves reader behaviour None Not earned A specific candidate caused the harm Surgical series at floor Not earned Readers differ on identical bundles One same-bundle observation Exploratory Context helps in real coding sessions No ecological evidence Not earned Live runtime context improves behaviour Second live wave, transport verified: no measured difference among none, distractor and oracle, all at floor Not earned; a null on the measured variables Next experiments The following experiments are proposals, not completed work. Exact reader transfer would hold task, prompt bytes, bundle bytes, decoding settings and scoring fixed while changing only a pinned reader identity, with matched repeats. Calibration repair would resolve the target/value mismatch, publish a new fixture version and first demonstrate that the reader can satisfy the intended no-context control. Candidate attribution would select a task and outcome with room to move, compare the parent bundle with one-item removal, exact restoration, volume-matched replacement and false-value replacement, and verify the constructed payloads independently. Discriminating assembly fixtures would freeze tasks where the compiler and hard-gated greedy produce meaningfully different legal bundles because of dependency closure, coupling, conflict preservation, alternatives or render repair, rather than assuming a different digest implies a consequential difference. Live behavioural qualification would require a current adapter canary, session and model attribution, a live no-context observation, raw-response retention, controlled decoding, and a grader that passes calibration before scoring experimental effects. Report planned, executed, transport-valid, behaviour-valid and excluded runs by condition; a transport failure is excluded from causal interpretation rather than counted as a treatment failure. A proposed experiment: multi-turn contamination and recovery Breunig’s context-failure taxonomy and Laban et al.’s multi-turn results motivate an experiment this chapter’s design can accommodate. The experiment is owned by Chapter 15 (which governs persistent generated state), and this chapter describes how it reaches behavioural evaluation. Consider a coding agent investigating a failing test. Initially it receives incomplete information and incorrectly concludes the cache is responsible. Later, a test result establishes the actual problem is database connection pooling. The final evidence is sufficient to identify the correct cause. The central comparison tests epistemic status as an intervention, holding other variables constant: Condition Context provided A — Full history All messages, incorrect assumptions, earlier attempts and new evidence B — Requirements consolidated Current task, accumulated user requirements, decisive environment observations; no earlier assistant answers C — Pruned history Original history with rejected attempts removed; decisive evidence preserved D — Typed state Observations, hypotheses, decisions and their evidential status separated; identical claims to B/C but with explicit contradicted status on the false hypothesis E — Oracle Minimum evidence required to solve the task correctly Controls applied to all non-ablated conditions: Evidence equivalence. Every condition receives the same decisive environmental observations and current requirements, verified independently of the model. Status ablation. Condition D carries the same false hypothesis as the relevant baseline but with an explicit contradicted status; wording and position are held stable where possible. Representation ablation. A separate arm compares free-form and typed representations containing the same claims, evidence references and statuses. Budget sensitivity. Both natural-size comparisons and matched-budget controls are included, so reducing interference isn’t mistaken for a benefit unique to typed state. Frozen outcomes. The correct diagnosis, required next action, prohibited repetitions and acceptance criteria are defined before generating model responses. Measure correct diagnosis, reuse of the rejected hypothesis, next-action correctness, repetition of prohibited actions, tool calls and trajectory tokens, with predeclared grading rules. However, Conditions A through E in the table vary history length, omitted assumptions, representation, status labels and available context together. They are therefore useful as whole-policy comparisons, not a clean test of epistemic status alone. To isolate status, create a matched pair from one frozen candidate set containing the same false hypothesis and decisive evidence: change only an externally supplied contradicted status while holding wording, order, representation and surrounding context fixed. Test representation in a separate factorial contrast, then examine matched-budget and natural-size variants. If the status-only arm does not improve recovery on discriminating tasks, a benefit from the broader typed-state condition cannot be attributed to its status field. Falsifiers. The influence claims weaken if recomputation shows miscounts or if matched runs differed beyond the bundle. The utility claims weaken if the grader is not deterministic or truth changed after outputs. The harm claim weakens if labels were assigned after seeing outputs. The compiler null weakens if a scoring error hid an action difference. Reader dependence stays unearned while readers see different bundles. What the evidence amounts to Changing context can change a model’s response, and response change is not utility. On six constructed tasks, moving from no supplied context to the compiler’s frozen bundle changed every full response and four of six decisions, improved two scored outcomes, worsened one and left three unchanged. The compiler’s extra machinery produced no decision-level or score-level difference from hard-gated greedy on these fixtures, and the hidden-evidence oracle none from the compiler. One task showed the more important failure: every supplied bundle moved the reader from a hold to a harmful release, and the surgical variants could not identify a responsible candidate. A control failed for a reason of its own, and a second-reader probe confounded reader and bundle. These define the reach of the evidence. The supported conclusion is narrower than a general claim that better context helps. On six constructed tasks, the recorded no-context and compiler-bundle calls differed in all six full responses and four actions; task scores improved twice, worsened once and tied three times. The reader’s immutable identity was not preserved, the calibration task exposed a grading-contract defect, and the item-level surgical study began at a floor. Separately, the second live wave verified context conditions at an earlier host boundary but observed no difference on its recorded metrics, all of which remained at a score floor of zero. Thus the experiments show both that context conditions can be associated with helpful and harmful task outcomes in the constructed ladder and that verified delivery is insufficient to demonstrate benefit. They do not establish production prevalence, a behavioural advantage for the compiler, candidate-level causation or cross-reader generality. Those are the questions the next controlled and ecological studies must earn. What does the finished system look like, and what is still open? References Project Context evidence register (companion repository, evidence/README.md). Consumed here: Matched behaviour, Reader-transfer probe and Instrumentation failure. Real reader behaviour on constructed tasks, the main reader recorded only by a moving alias; no ecological claim. Li, H., Zhu, L., Zhang, B., et al. “ContextBench: A Benchmark for Context Retrieval in Coding Agents.” Preprint, arXiv:2602.05892, February 2026. https://arxiv.org/abs/2602.05892 Gloaguen, T., Mündler, N., Müller, M., Raychev, V., Vechev, M. “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” Preprint, arXiv:2602.11988, February 2026, revised June 2026. https://arxiv.org/abs/2602.11988 Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., Schütze, H. “NoLiMa: Long-Context Evaluation Beyond Literal Matching.” ICML 2025. https://arxiv.org/abs/2502.05167 Zhu, J., Wu, J., Hu, M., et al. “SWE Context Bench: A Benchmark for Context Learning in Coding.” Preprint, arXiv:2602.08316, February 2026. https://arxiv.org/abs/2602.08316 Laban, P., Hayashi, H., Zhou, Y., Neville, J. “LLMs Get Lost In Multi-Turn Conversation.” arXiv:2505.06120, May 2025. Preprint. https://arxiv.org/abs/2505.06120 Breunig, D. “How Long Contexts Fail.” 22 June 2025. Practitioner synthesis; four failure modes and mitigation taxonomy. https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html Breunig, D. “How to Fix Your Context.” 26 June 2025. Practitioner synthesis; tool loadout, context quarantine, pruning, summarization, offloading. https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.html