Part V — Debugging AI-Assisted Development and Research

The contract was clear — the agent never saw the files

Chapter 25 pinned intent. Now the agent with a perfect contract edits the wrong module, reimplements an existing helper, and misses the migration the contract names by path. The intent artifact is innocent. The working set is the suspect.

OBSERVATION: the agent’s session log shows 4 files read; the fix required 7, including the migration and the helper the agent duplicated. HYPOTHESIS H1 (missing-file): the agent never received the decisive file — retrieval omitted it or the working set excluded it. H2 (misread-file): the file was in context but the agent’s edits contradict its content — present but unused. H3 (stale context): the file was read but an older revision — edits fit a version that no longer exists. INFERENCE: none yet — H1/H2/H3 predict different working-set dumps and are separable only by one.

This chapter’s question: what did the agent actually see — and is the failure missing context, misread context, or stale context?

Why “paste the file into the prompt” fails first

The obvious move — pasting the missing file and re-running — fails because it confounds three mechanisms and teaches nothing durable. Four defects hide behind the paste-and-pray loop:

1. Confounded repair. Pasting changes content, ordering, working-set size, and retrieval all at once. Success proves nothing about which gap mattered; the next session fails identically.
2. Recency theater. The pasted file lands at the prompt’s end, gets maximal attention, and the fix works — until the same file retrieved mid-context in production is ignored again. Position effects masquerade as comprehension.
3. Stale-copy resurrection. The pasted copy is the engineer’s checkout; the agent’s index holds an older revision. The fix fits one revision and breaks the other.
4. Symptom closure. “It works now” closes the ticket without a working-set record. No prevention artifact survives: no retrieval assertion, no context manifest, no staleness check.

OPINION: context is the agent’s visual field. Debugging an agent without its working-set dump is debugging a driver without knowing whether the windshield was painted shut.

The mental model: context forensics — reconstruct the agent’s observable world before judging its decisions. Every edit is rational relative to some working set; find that set first, and the “stupid” edit usually becomes an inevitable one.

The leverage is real. Xia and colleagues’ Agentless study found that a plain three-phase pipeline — localize the right files, repair, validate — matched or beat complex tool-using agents on SWE-bench, with hierarchical localization (files, then classes and functions, then lines) doing most of the work (Xia et al., 2025). If finding the right files is what separates success from failure, then the working-set dump is not a diagnostic afterthought; it is the primary evidence.

The method: the working-set dump and the three-way separation

Demand the session’s observable world as artifacts, not recollections:

1. Dump the working set. Every file the agent read (path + content hash + bytes actually sent, not the file as it exists now), every retrieval result (query, rank, top-k, returned IDs), every tool return the agent consumed, in order. Truncation points included — a file “read” but truncated at 8K tokens was half-seen.
2. Separate the three gaps. For each file the fix required: absent from the dump → H1 (missing-file); present with hash matching current revision but contradicted by edits → H2 (misread-file); present with hash matching an older revision → H3 (stale context).
3. Run the single-variable context intervention. Add (H1), highlight/reorder (H2), or refresh (H3) exactly one file’s context status; hold prompt, model, seed, and all other files fixed; re-run ≥3 trials.

    flowchart TD
    F["for each file the correct fix requires"] --> D{"present in the working-set dump (bytes actually sent)?"}
    D -->|no| H1["H1 missing-file — retrieval rank below top-k / outside the search frontier; fix retrieval"]
    D -->|yes| H{"sent-bytes hash matches the current revision?"}
    H -->|"no, older revision"| H3["H3 stale context — fix cache / invalidation"]
    H -->|yes| T{"truncated, or edits contradict its content?"}
    T -->|"truncated"| H1t["half-seen — treat as H1 for the cut span"]
    T -->|"present, current, still ignored"| H2["H2 misread-file — surface + reorder (often position, not comprehension)"]
    H1 --> I["single-variable intervention: add (H1) / surface (H2) / refresh (H3) ONE file, >=3 trials"]
    H2 --> I
    H3 --> I
  

WORKING-SET DUMP (per required file):
path | bytes sent (hash) | current rev (hash) | match? | retrieval rank/top-k | truncated? | verdict
db/migrate_042.py | — (never sent) | h:9f2c | N/A | rank 47, top-k 5 | — | H1 missing-file
lib/helpers.py | h:11ab (12KB sent) | h:11ab | YES | direct read | no | H2 misread-file
api/routes.py | h:77e0 (sent) | h:3bd9 | STALE | cached snapshot | — | H3 stale context
RULE: one row, one verdict. Absent / present-matching / present-stale predict
different single-variable repairs; any combined repair is confounded.

OBSERVATION (constructed illustration, not a measured run): the dump showed the migration never sent (retrieval rank 47 below top-k 5 → H1), the helper sent whole and current yet duplicated (→ H2), and routes sent at a stale hash (→ H3). Three files, three different gaps, one session. UPDATED BELIEF: H1/H2/H3 each supported for different files in this instance — context failure is per-file, not per-session. A session-level verdict (“the agent can’t read”) would have mistargeted all three repairs.

Example: the duplicated helper, forensically separated

Contract names lib/helpers.py::rate_key(); the agent writes its own make_key() and imports nothing. Forensics before theorizing:

# context forensics: reconstruct, then separate (nothing re-run yet)
required = ["lib/helpers.py", "db/migrate_042.py", "api/routes.py"]
dump = working_set_dump(session_log)  # path -> (bytes_sent_hash, truncated?, rank)
for path in required:
    sent, current = dump.get(path), hash_file(path)
    if sent is None:
        print(path, "H1 missing-file — retrieval rank:", retrieval_rank(path))
    elif sent.hash != current:
        print(path, "H3 stale — sent", sent.hash, "current", current)
    else:
        print(path, "H2 candidate — present+current; check edits vs content")
# Discriminating intervention (one variable): H1 -> add the file alone, all else fixed;
# H2 -> same set, helper moved to prompt head with pointer comment; H3 -> refresh to
# current hash alone. Each x3 trials; prediction pre-written per hypothesis.

In the constructed case the helper row reads present-and-current: H1 and H3 exonerated for this file, H2 live. The H2 intervention (same files, helper surfaced with an explicit pointer, nothing added) is the only unconfounded next step — and if it fails across trials, the verdict is UNKNOWN (attention-ordering vs. comprehension-depth still confounded), not “the agent is broken.”

No self-report (“I read all the files”), confidence percentage, or single successful re-run substitutes for the dump. The agent’s account of its context is verbal behavior; the byte log is evidence (Chapter 3).

Research lineage: localization is the job, and retrieval is how it fails

Localization dominates agent outcomes. Beyond the headline result, Agentless’s design is an argument that most of an SE agent’s value is in getting the right code in front of the model — the repair step is comparatively easy once localization is right (Xia et al., 2025). And it fails often: on curated benchmarks, current SE-agents get the right file only about three-quarters to four-fifths of the time (up to ~93% with a model fine-tuned specifically for localization), so on roughly one issue in four to five the decisive file is never in front of the model (Chen et al., 2025). That is the H1 rate the “rule H1 out first” rule is responding to. Chen and colleagues’ LocAgent also names the mechanism that lifts localization: multi-hop reasoning over a repository dependency graph — files, classes, and functions linked by imports, calls, and inheritance — which raised downstream issue-resolution by about 12%. So H1 (missing-file) is the failure mode to rule out first; the retrieval-rank column in the dump is where it shows for a fixed retrieved set, and for an agent that opens files on demand the equivalent evidence is how far the file sat from the agent’s search frontier.

Repo-level retrieval is decisive and in-file context is not enough. Zhang and colleagues’ RepoCoder showed that iteratively retrieving relevant code from across the repository — using the generation so far as the next query — improves completion by over 10% against an in-file baseline that has no cross-file context at all (Zhang et al., 2023). When the dump shows a required file at rank 47, that is not an edge case; it is the default failure of a weak retriever.

H2 is often position, not comprehension. A required file that is present, current, and still ignored is frequently a lost-in-the-middle case (Chapter 18): the same content surfaced at the top of the context gets used. That is exactly why the H2 intervention surfaces and reorders rather than re-adds.

Some “agent failures” are underspecified issues. Agentless’s authors manually re-graded SWE-bench Lite and set aside instances with insufficient or misleading issue text as SWE-bench Lite-S. Before convicting the working set, confirm the contract (Chapter 25) actually names what the fix requires — an H1 verdict on a file the issue never mentioned is really an intent-gap.

Lab 26: context forensics with competing file-level hypotheses (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own dump table.

Setup. Take one failed coding-agent session with logs (or instrument a fresh one: fixed prompt + contract, pinned model/seed, retrieval top-k logged, file reads hashed). Freeze the session log, file revisions, and hashes. The context status of one file is the independent variable; prompt, model, seed, and all other files are controlled.

Task.

1. Before dumping, write H1/H2/H3 with distinct predicted dump rows: H1: “required file absent from dump (rank below top-k or never read)”; H2: “present, hash-matching, edits contradict content”; H3: “present, hash-mismatched to an older revision.”
2. Build the working-set dump table for every file the correct fix touches. Label each row H1/H2/H2-candidate/H3/UNKNOWN with hashes and truncation flags.
3. Run one single-variable context intervention on one file (add / surface / refresh), ≥3 trials, all else fixed. Record OBSERVATION (per-trial outcomes verbatim) and UPDATED BELIEF. Multi-file repairs in one run are explicitly confounded — record as such, not as evidence.

Success criterion. A completed dump table with per-file verdicts plus one single-variable intervention with per-trial results. A pasted-file success story without the dump is explicitly not completion.

Companion tool: Context Map Inspector

What it accepts: the session log (reads, retrieval queries/ranks, tool returns, truncation points), current file hashes, and the list of files the correct fix requires. What it performs: it reconstructs the working set as actually sent (hashes + truncation flags), diffs sent vs. current revisions, labels each required file missing/misread-candidate/stale, blocks session-level verdicts while any required row is UNKNOWN, and requires ≥3 trials per context intervention. What it can establish: whether a required file was absent, present-but-contradicted, or stale — and which single-variable repair the evidence licenses, for the examined session only. What it cannot establish: why present-and-current content was ignored (attention vs. comprehension needs the H2 intervention series), future retrieval reliability, or anything about files outside the required set. It never treats self-reports, confidence, agreement, or single-run repair as diagnosis. How its output changes your next action: H1 routes to retrieval repair (top-k, query, indexing freshness, and dependency-graph expansion — pulling in the files structurally adjacent to the ones the contract names); H2 routes to surfacing/ordering interventions; H3 routes to cache/invalidation repair; UNKNOWN routes to re-instrumentation, never to verdict.

Paper form, sufficient for this chapter:

Session: ___  Model/seed: ___ / ___  top-k: ___
Required files: ___ (n=___)
Per file: path ___ | sent ___ | current ___ | match/STALE/absent | rank ___ | trunc Y/N | verdict H1/H2/H3/UNKNOWN
INTERVENTION (one file, one change): ___  TRIALS ×3: ___ ___ ___  UPDATED BELIEF: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Dump before diagnosis.

Reusable procedure: forensics before repair

1. Freeze the session — logs, file revisions, hashes, retrieval parameters.
2. List required files — what the correct fix touches, from the contract outward.
3. Dump the working set — sent bytes per file, hashes, ranks, truncation flags.
4. Label per file — missing (H1) / present-contradicted (H2) / stale (H3) / UNKNOWN.
5. Intervene singly — one file, one change, ≥3 trials, all else fixed.

Failure modes

• Paste-and-pray. Adding files without the dump. Repairs confound content with position with set size.
• Session-level verdicts. “The agent doesn’t read code.” Per-file gaps need per-file repairs; global verdicts mistarget all of them.
• Self-report trust. “The agent says it read the file.” Verbal behavior is not a byte log (Chapter 3).
• Truncation blindness. “Read” logged while 70% of bytes were cut. A half-seen file is a quarter-understood one; check truncation flags.
• Staleness neglect. Debugging content while the defect is revision. Hash the sent bytes, not the current file.
• Single-trial repair claims. One successful re-run after pasting proves the repair. Nondeterministic agents need repeated trials or UNKNOWN.

Limits, per contract: one dump diagnoses one session under one revision set; it does not certify retrieval, does not explain attention internals, and does not transfer across prompts or index rebuilds. UNKNOWN where logs lack hashes or truncation records.

References

• Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering (FSE), 2025 (arXiv 2024). https://arxiv.org/abs/2407.01489
• Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. Proceedings of EMNLP, 2023, pp. 2471–2484. https://doi.org/10.18653/v1/2023.emnlp-main.151
• Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. LocAgent: Graph-Guided LLM Agents for Code Localization. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025, pp. 8697–8727. https://arxiv.org/abs/2503.09089
• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173. https://doi.org/10.1162/tacl_a_00638

Debugging Checklist

• Session frozen (logs, revisions, hashes, retrieval params)?
• Required-file list derived from the contract (not from the agent’s story)?
• Working-set dump complete (sent hashes, ranks, truncation flags per file)?
• Each required file labeled H1/H2/H3/UNKNOWN (per-file, never session-global)?
• Contract (Ch 25) confirmed to actually name the required files before an H1 verdict?
• H2 intervention surfaces/reorders (not re-adds) — position vs. comprehension?
• H1/H2/H3 FORECASTs pre-written with distinct predicted dump rows?
• Single-variable context intervention (one file, ≥3 trials, all else fixed)?
• No self-report, confidence, or single-run repair cited as diagnosis?

What This Chapter Established

• Context forensics for coding agents: the working-set dump with the missing-file vs. misread-file vs. stale-context separation, demonstrated on the constructed duplicated-helper case.
• The single-variable context intervention (add / surface / refresh one file, ≥3 trials) with pre-written per-file forecasts; no measured runs claimed.
• Lab 26 as a proposed dump record the reader executes; the Context Map Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any claim about attention mechanisms, any retrieval-quality benchmark, or any general agent capability statement. One session mapped; nothing universal.
• Research grounding: localization (getting the right files in front of the model) is the dominant factor in SE-agent success (Agentless / Xia et al.), and it misses often — current agents get the right file only ~75–80% of the time on curated benchmarks, up to ~93% with a dedicated localization model, and lifting localization via repo-dependency-graph reasoning lifted issue resolution ~12% (LocAgent / Chen et al.) — so the working-set dump is primary evidence; repo-level retrieval is decisive and in-file context alone is weak (RepoCoder); H2 is often position, not comprehension (lost in the middle); and some “context” failures are really underspecified contracts (SWE-bench Lite-S) — check Chapter 25 first.
• Forward link: context now verified present and current — yet the agent’s architecture proposal still violates the latency budget. Seeing the files was necessary but not sufficient. Designs are Chapter 27’s jurisdiction.

Next

The agent saw everything and still proposed a system that cannot meet its constraints — three fan-out reads per request against a 200 ms budget, eloquently defended. Context forensics is clean; the design is not. Chapter 27, “Debugging AI-Generated Designs,” evaluates proposals against constraints, not eloquence: budgets, tradeoff tables, and the rejected alternatives the design owes you.