Part VIII — Building the AI Debugger

The failure that could not be reopened

Chapter 44 permitted the AI assistant to propose — but only from a frozen, hashed bundle. That bundle did not yet exist as a defined artifact. Here is why it must. A practitioner investigates a support chatbot that quoted a refund policy nobody can find. She screenshots the answer, copies the prompt from memory, and asks the team to reproduce it. Three days later nobody can: the model version has rolled forward, the retrieved documents have been re-indexed, the temperature was never recorded, and the “prompt” she saved differs by two sentences from what was actually sent. The failure is real, witnessed — and undebuggable.

OBSERVATION: the saved material (screenshot + recalled prompt) omits model revision, retrieval snapshot, parameters, and tool/environment versions; re-execution under partially matched conditions produces a different answer. HYPOTHESIS H1 (prompt defect): the wording caused the fabrication. H2 (retrieval defect): the evidence set caused it. H3 (version drift): the model or index changed under the investigation. INFERENCE: none yet — H1/H2/H3 are indistinguishable without the frozen inputs, because reproduction quality sets the ceiling on diagnosis quality and there is no reproduction here.

This chapter’s question: what is the minimal sufficient bundle — the crash-dump analog — that makes an AI failure reopenable by anyone, including an AI assistant, after the fact?

Why “save the prompt and the answer” fails first

The obvious move — keeping the prompt text and the output — fails because an AI execution is not a function of the prompt alone. Five omissions destroy reopenability:

1. Unpinned model. Prompt + answer without model name, revision, and decoding parameters describes no executable thing. A version roll is a different system wearing the same name.
2. Unsnapshotted context. Retrieved chunks cited by ID but not preserved verbatim with index version. Re-indexing silently swaps the evidence under a stable-looking prompt.
3. Missing tool records. Tool calls paraphrased (“it checked the policy”) instead of recorded with arguments, versions, and verbatim returns. The trajectory’s middle is gone.
4. No artifact hashes. Nothing content-hashed, so nobody can verify that today’s “same” prompt, document, or config is byte-identical to the failure’s. Drift becomes invisible.
5. Answer-only preservation. The final text kept, intermediates discarded. First-divergence localization needs ordered intermediates; the end of the chain convicts nothing.

OPINION: a screenshot of an AI failure is a postcard from a crash site after the wreckage was cleared. Debuggable means rebuildable.

This mismatch is a documented, decades-old problem. Bettenburg and colleagues surveyed 466 developers and reporters across Apache, Eclipse, and Mozilla and found that the items developers most need — steps to reproduce, stack traces, test cases — are exactly the items reporters find hardest to supply, and that incomplete reports measurably lengthen fix time (Bettenburg et al., 2008). The seven-slot dump is a bug-report schema for AI failures, and the screenshot is the classic under-informative report.

The mental model: the crash dump is a frozen executable past — everything needed for a third party (human or assistant) to re-execute the failing behavior deterministically-as-possible and to verify byte-identity of every input. Native crash dumps freeze registers and memory; the AI dump freezes prompt, context, parameters, artifacts, and versions — each with a hash, each with a source.

The method: the seven-slot bundle with hash discipline

A complete AI crash dump has seven slots. Every slot is either filled with hashed content or explicitly marked ABSENT (which downgrades what can be claimed — absence is information, not embarrassment):

1. Prompt record. Exact sent text (system, developer, user, tool preamble), verbatim, with a content hash. Recalled or retyped prompts are marked RECONSTRUCTED and cap all claims at weak.
2. Context snapshot. Every retrieved/injected document verbatim with source ID, index/corpus version, and per-item hashes — plus the retrieval query, ranker version, and full ranked list (not just top-k), because exclusion is evidence too.
3. Parameters. Model name + revision (pinned; vendor version strings are changeable facts — record the exact string and date), decoding settings (temperature, top-p, seed, max tokens), and stop/tool-config. Unrecorded temperature → nondeterminism UNKNOWN.
4. Trajectory. Ordered tool calls with arguments, tool versions, verbatim returns, and timestamps/sequence numbers — the Chapter 37 six-field discipline, referenced not repeated.
5. Artifacts. System configs, policy files, eval harnesses, and code revisions involved, each hashed. “The policy file” without a hash is a rumor.
6. Environment. Runtime versions, feature flags, time of run, region/endpoint where relevant (changeable facts, attributed and dated).
7. Outcome. The observed output verbatim with hash, the intended behavior written as measurable expectation, and the first-divergence pointer if already localized (or UNKNOWN if not).

    flowchart TD
    C["capture at failure time, verbatim — never from memory"] --> S1["1 prompt: exact sent text + hash"]
    C --> S2["2 context: every retrieved doc verbatim + index version + full ranked list + hashes"]
    C --> S3["3 params: model revision string + date, temperature, top-p, seed, stop config"]
    C --> S4["4 trajectory: ordered tool calls, versions, verbatim returns (Ch37 six fields)"]
    C --> S5["5 artifacts: configs, policy files, eval harness, code revs — each hashed"]
    C --> S6["6 environment: runtime versions, feature flags, run time, region"]
    C --> S7["7 outcome: output verbatim + hash, intent as a measurable expectation, first-divergence pointer or UNKNOWN"]
    S1 --> M{"every slot PRESENT + hash-verified?"}
    S2 --> M
    S3 --> M
    S4 --> M
    S5 --> M
    S6 --> M
    S7 --> M
    M -->|"slot ABSENT or RECONSTRUCTED"| D["mark it; downgrade the strongest claim the bundle supports (Ch49 sets the bar)"]
    M -->|"all present"| RO["independent reopen test: a third party re-executes from the manifest alone, x3 if nondeterministic"]
  

CRASH DUMP MANIFEST (worked sketch; constructed, not a measured run):
case refund-quote-041 | dump v1 | created 2026-__-__ (date pinned at capture)
[1] prompt ........... sha:9f2c… verbatim (3 msgs) ............ PRESENT
[2] context .......... 6 chunks, index v412, hashes ok ........ PRESENT
[3] params ........... model rev ___ temp 0.2 seed 7 .......... PRESENT
[4] trajectory ....... 4 tool calls, returns verbatim ......... PRESENT
[5] artifacts ........ policy.json sha:44ab… .................. PRESENT
[6] environment ...... runtime ___ flags ___ .................. PRESENT
[7] outcome .......... output sha:c001… intent written ........ PRESENT
ABSENT: none. REOPENABILITY: full — third party can re-execute.
RULE: any slot ABSENT or RECONSTRUCTED downgrades the strongest
claim the bundle supports (Ch49 sets the exact bar).

OBSERVATION (constructed illustration, not a measured run): the complete manifest above supports single-variable re-execution; the screenshot-only version from the opening supports none of H1/H2/H3. UPDATED BELIEF: H1/H2/H3 remain live in the screenshot case (undebuggable, not exonerated); in the full-bundle case each predicts a distinct re-execution signature and becomes testable.

One record, filled over time. The seven-slot dump is the book’s single diagnostic-case record, frozen at capture time before hypothesis work begins. The same record appears elsewhere with more fields filled as diagnosis proceeds — Chapter 1’s hypothesis record, Chapter 24’s AI work-product case file, Chapter 34’s evidence ledger, Chapter 36’s trajectory, Chapter 49’s preregistered ledger, Chapter 59’s incident packet. They share a skeleton: intent as a measurable expectation, the frozen inputs, the ordered trajectory, the evidence with provenance, competing hypotheses with predictions, the first-divergence pointer, the disposition. Capture it once here; the later chapters extend it, they do not re-derive it.

No assistant summary of the bundle, no confidence in its completeness, no agreement between two readers that “this looks complete,” and no downstream relief substitutes for hash-verified slots. Manifests, not impressions.

Example: building the dump for the phantom refund quote

The practitioner re-captures properly: exact prompt from the request log (not memory), six retrieved chunks verbatim with index v412, model revision string and temperature 0.2 with seed 7, four tool calls with verbatim returns, policy.json hashed, output hashed, intent written as “quote must cite a policy section present in the snapshot, or abstain.” The dump convicts nothing by itself — but when H2 is tested (re-execute with chunk 4 removed), the quote changes as predicted while H1’s wording-swap changes nothing across ≥3 trials. The bundle did not diagnose; it made diagnosis possible. That is its entire job.

Research lineage: the crash dump has a systems precedent

Automated failure capture at scale is a solved discipline. Glerum and colleagues describe Windows Error Reporting: minidumps captured automatically from a billion machines, a progressive collection strategy (minimal data by default, detailed data escalated on demand), automated bucketing of similar failures, and — the part that matters for later chapters — using the statistics across dumps as a debugging tool to find bugs invisible at single-report scale (Glerum et al., 2009). The seven-slot manifest is a minidump for AI failures; the ABSENT-slot discipline is progressive collection made honest; and Part VIII’s later aggregation of dumps is WER’s statistics-at-scale.

“What to record about the model” is already standardized. The parameters and artifacts slots overlap with the model-card and datasheet movement — Mitchell and colleagues’ Model Cards and Gebru and colleagues’ Datasheets for Datasets specify the version, training, and provenance metadata that make an ML artifact interpretable after the fact (Mitchell et al., 2019; Gebru et al., 2021). The params, artifacts, and environment slots are converging further with W3C PROV-compliant provenance models for ML pipelines, full-stack reproducibility-metadata capture, and AI Bill of Materials proposals, with the AI Incident Database as the public failure-report side (McGregor, 2021). A crash dump that cites “the model” and “the policy file” without those identifiers is missing metadata the field is standardizing.

The Crash Dump Builder is a report-quality checker. Bettenburg’s team shipped CUEZILLA, a tool that scores a new bug report and recommends which missing elements would raise its quality. The companion tool here does the same job for the seven slots.

Lab 45: capture a dump and grade its reopenability (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own manifest.

Setup. Take one AI failure you can trigger (prompt, RAG, or agent step). The capture discipline (ad-hoc notes vs. this chapter’s seven-slot manifest) is the independent variable; the failure and system revisions are controlled. A second person (or your future self after ≥48h) is the reopener.

Task.

1. Before capturing, write H1/H2/H3 with distinct predicted reopenability outcomes: H1: “seven-slot dump lets the reopener re-execute and reach the same first divergence”; H2: “ad-hoc notes suffice for the same divergence”; H3: “neither suffices because the failure is nondeterministic without recorded seeds.”
2. Capture both forms; hand only the artifact to the reopener (no verbal context).
3. The reopener records per-slot PRESENT/ABSENT/RECONSTRUCTED, verifies hashes, and attempts re-execution ≥3 trials where nondeterministic.

Success criterion. A manifest with per-slot hashes and ABSENT markings, a reopener’s independent re-execution log, and a written downgrade statement for every absent slot. An unreopened dump is explicitly not completion.

Companion tool: Crash Dump Builder

What it accepts: the failing case pointer, raw logs/configs/retrieval snapshots the stack can export, and the written intent statement. What it performs: it assembles the seven slots, hashes every item, marks unexportable slots ABSENT with the resulting claim-downgrade, stamps versions and dates (flagging changeable facts), and emits the manifest above. What it can establish: whether a failure is frozen completely enough to reopen, and which slots limit the strongest supportable claim — for the examined case only. What it cannot establish: the cause, generality across cases, or future reproducibility under changed versions. It never treats paraphrase, screenshots, confidence, agreement, single re-runs, or symptom relief as bundle contents. How its output changes your next action: manifest complete → proceed to hypothesis work (Ch47) and experiment design (Ch48); slots ABSENT → instrument the missing export first; RECONSTRUCTED prompt → cap all downstream claims at weak until re-captured live.

Paper form, sufficient for this chapter:

Case: ___  Dump v___  Captured: ___ (by ___)
SLOTS (PRESENT/hash | ABSENT | RECONSTRUCTED):
prompt ___ | context (items ___, index ___) ___ | params (model ___ temp ___ seed ___) ___
trajectory (calls ___) ___ | artifacts ___ | environment ___ | outcome ___
REOPEN TEST: reopener ___ trials ___ divergence match? ___  DOWNGRADE: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Freeze the past or forfeit the diagnosis.

Reusable procedure: freeze every AI failure the same way

1. Capture verbatim — prompt, context, returns, output; never retyped, never summarized.
2. Pin versions — model, index, tools, runtime; exact strings with dates.
3. Hash everything — content hashes per item; verify before handing off.
4. Mark absence — ABSENT/RECONSTRUCTED slots with explicit claim-downgrades.
5. Test reopening — a second party re-executes from the manifest alone, ×3 where nondeterministic.

Failure modes

• Screenshot debugging. Keeping the answer and discarding the execution. Answers are symptoms; executions are evidence.
• Recalled prompts. “The prompt was roughly…” Roughly is RECONSTRUCTED, and RECONSTRUCTED caps everything.
• Unversioned retrieval. Chunk IDs without index version. IDs are pointers; pointers rot.
• Hashless bundles. All text present, nothing verifiable. Without hashes, drift is undetectable.
• Silent ABSENT slots. Missing parameters nobody marks. Unmarked absence inflates claims; marked absence disciplines them.
• Dump-as-diagnosis. Treating completeness as correctness. A perfect dump with no hypothesis work is a well-preserved mystery.

Limits, per contract: one manifest covers one case under one version set; it warrants reopenability, not cause; any version roll, re-index, or flag flip after capture voids re-execution claims until re-captured. UNKNOWN wherever a slot is ABSENT or a hash fails to verify.

References

• Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. What Makes a Good Bug Report? Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2008, pp. 308–318. https://doi.org/10.1145/1453101.1453146
• Kirk Glerum, Kinshuman Kinshumann, Steve Greenberg, Gabriel Aul, Vince Orgovan, Greg Nichols, David Grant, Gretchen Loihle, and Galen Hunt. Debugging in the (Very) Large: Ten Years of Implementation and Experience. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP), 2009, pp. 103–116. https://doi.org/10.1145/1629575.1629586
• Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT), 2019, pp. 220–229. https://doi.org/10.1145/3287560.3287596
• Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for Datasets. Communications of the ACM 64(12), 2021, pp. 86–92. https://doi.org/10.1145/3458723
• Sean McGregor. Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. Proceedings of the AAAI Conference on Artificial Intelligence, 2021 (arXiv:2011.08512). https://arxiv.org/abs/2011.08512

Debugging Checklist

• Exact prompt captured verbatim with hash (or marked RECONSTRUCTED)?
• Full context snapshot with index version and per-item hashes?
• Model revision, decoding params, seed recorded as exact strings?
• Trajectory with tool versions and verbatim returns?
• Artifacts and environment pinned and hashed?
• Output verbatim with written intent and divergence pointer?
• Every ABSENT slot marked with its claim-downgrade?
• Independent reopen tested from the manifest alone (×3 if nondeterministic)?

What This Chapter Established

• The AI crash dump: the seven-slot minimal sufficient bundle (prompt, context, params, trajectory, artifacts, environment, outcome) with hash requirements and ABSENT/RECONSTRUCTED discipline — demonstrated on the constructed phantom-quote case, no measured runs claimed.
• The freeze-list schema and manifest form that Chapter 44’s delegation gates consume.
• Lab 45 as a proposed capture-and-reopen record the reader executes; the Crash Dump Builder contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any causal verdict, any cross-case generality, or any reproducibility under changed versions. One schema defined; nothing diagnosed.
• Research grounding: the seven-slot dump is a bug-report schema for AI failures, and the screenshot is the classic under-informative report that lengthens fix time (Bettenburg et al.); it is a minidump with progressive collection (Windows Error Reporting / Glerum et al.), whose statistics-at-scale Part VIII later exploits; the params/artifacts slots restate the model-card / datasheet metadata the field already agreed is mandatory (Mitchell et al.; Gebru et al.) and are converging with W3C PROV / AI-BOM provenance standards (AI Incident Database as the failure-report side — McGregor); the Crash Dump Builder is a CUEZILLA-style report-quality checker.
• One diagnostic-case record: the seven slots are that record frozen at capture; Ch1 (hypothesis record), Ch24 (case file), Ch34 (evidence ledger), Ch36 (trajectory), Ch49 (ledger), Ch59 (packet) are the same object with more fields filled as diagnosis proceeds.

Next

Failures can now be frozen — but nothing yet forces the assistant (or the practitioner) to obey the book’s rules when working from the bundle. Frozen evidence with undisciplined consumers still produces narration. Chapter 46, “Diagnostic AI Invariants,” encodes the Chapters 01–03 rules as machine-checkable invariants the debugging assistant itself must satisfy.