Debugging AI on Programmer.ie: Modern AI programming

What Does It Mean to Debug?

Sun, 06 Sep 2026 21:00:00 +0000

Part I — Debugging From First Principles

The puzzle: two fixes, one bug

A team owns a small billing function. Invoices above €1,000 get a 10% discount. One Monday, a €1,200 invoice goes out at full price. Two developers investigate independently.

Developer A reads the code, spots a suspicious comparison, flips < to <= somewhere that “looks wrong,” reruns the failing invoice, and sees €1,080. Fixed. She closes the ticket.

The First Divergence

Sun, 06 Sep 2026 21:00:00 +0000

Part I — Debugging From First Principles

The puzzle: the report is wrong, but where?

Chapter 1 defined debugging as constructing the smallest evidence-backed causal account that predicts the failure and its reversal — pursued, inside a pinned envelope, by finding the earliest transition surviving forward and reverse intervention — and split “earliest divergence” into three levels: first difference, first relevant difference, first causal divergence. This chapter makes the localization part executable — how you get to a candidate first divergence efficiently, before the hypothesis cycle promotes it to a cause.

Evidence Before Explanation

Sun, 06 Sep 2026 21:00:00 +0000

Part I — Debugging From First Principles

The puzzle: the model explains, fluently, and is wrong

Chapters 1 and 2 assumed your checkpoints record reality. This chapter removes that comfort.

Scenario. A RAG assistant answers: “Your refund was processed on September 2; reference RB-8814.” Asked how it knows, it replies: “I retrieved ticket #4471 and the refund ledger, which both confirm RB-8814.” Confident. Cited. Coherent.

The ledger says no such thing. Ticket #4471 is about a different customer. Reference RB-8814 does not exist in any retrieved document. Every sentence of the explanation is fluent and false.

The Debugging Stack

Sun, 06 Sep 2026 21:00:00 +0000

Part I — Debugging From First Principles

The puzzle: three suspects, one `KeyError`

Chapter 3 taught you to trust only frozen artifacts. This chapter tells you where to look for them — in what order, and at which layer.

Scenario. A batch job crashes:

KeyError: 'refund_id'
  File "report.py", line 90, in build_row
    row["refund_id"] = order["refund_id"]

Three engineers propose three fixes within minutes:

Engineer A: “Line 90 assumes a key that clean doesn’t guarantee. Guard with .get().” (Code layer.)
Engineer B: “The upstream export dropped the column yesterday. Re-fetch the data.” (Data layer.)
Engineer C: “Works on my machine — the job container pins pandas 1.5, prod runs 2.x, and the CSV reader changed dtype inference.” (Environment layer.)

OBSERVATION: order lacks refund_id at line 90 on this input, this code version, this container. HYPOTHESES: H1 code-assumption, H2 data-regression, H3 environment-drift. INFERENCE: none yet — all three predict the identical traceback.

Reading Python Exceptions

Sun, 06 Sep 2026 21:00:00 +0000

Part II — Debugging Deterministic Software

The crash that names the wrong culprit

A nightly invoice job dies at 02:14 with this:

Traceback (most recent call last):
  File "jobs/run_invoices.py", line 41, in <module>
    main()
  File "jobs/run_invoices.py", line 36, in main
    totals = summarize(orders)
  File "billing/totals.py", line 88, in summarize
    row["refund_id"] = order["refund_id"]
KeyError: 'refund_id'

The on-call engineer’s eyes lock onto line 88 of totals.py. That is where the traceback ends, so that must be where the bug lives. She adds .get("refund_id"), the job goes green, and three weeks later finance finds $41,000 in refunds silently missing from the report. The traceback told the truth. She read it backwards — or rather, she read only the last line and treated it as the diagnosis.

Inspect State, Don't Guess

Sun, 06 Sep 2026 21:00:00 +0000

Part II — Debugging Deterministic Software

The function that reads correctly and runs wrong

Chapter 5 convicted the handoff. Now you open the convicted function, and it looks innocent:

# billing/discounts.py
def apply_discount(order):
    rate = RATE_TABLE[order["tier"]]       # line 12
    total = order["subtotal"] * (1 - rate) # line 13
    if order.get("coupon"):                # line 14
        total -= COUPONS[order["coupon"]]  # line 15
    return round(total, 2)                 # line 16

A premium-tier order for $200 with a $20 coupon returns $160 instead of the expected $140. Read the code: line 12 looks up the tier rate (0.2 for premium), line 13 computes $160, lines 14–15 subtract the coupon. The logic reads correctly. Two engineers stare at it for twenty minutes and conclude “the coupon table must be stale” — then spend an afternoon auditing COUPONS, which is fine.

Debug the Boundary

Sun, 06 Sep 2026 21:00:00 +0000

Part II — Debugging Deterministic Software

The suite is green and the refunds are wrong

Chapter 6 taught you to inspect live state on the failing input. This chapter asks why the failing input was never run.

The paginate helper ships with 47 passing tests:

# billing/pages.py
def paginate(items, page_size):
    """Split items into pages of page_size. Returns list of pages."""
    pages = []
    for i in range(0, len(items), page_size):
        pages.append(items[i:i + page_size])
    return pages

Every test uses comfortable middles: 100 items with page size 10, 25 items with page size 5. Production runs it on the refund queue — and one night the queue holds exactly 0 items, then a single-item queue, then a queue whose length equals the page size exactly. The shown paginate itself returns [] correctly for the empty input (its range(0, 0, 10) never executes); the phantom page appears in the production path, where a chunk_count-driven sibling (convicted below) allocates one page too many at every exact multiple — including zero — and downstream code then indexes a page that should not exist. The single-item run works. All 47 tests stay green through all of it.

Assertions, Invariants, and Contracts

Sun, 06 Sep 2026 21:00:00 +0000

Part II — Debugging Deterministic Software

The corruption nobody heard

Chapters 5–7 kept finding breaks at the same place: the handoff. Now watch what happens when the handoff has no guard.

The clean → summarize pipeline from Chapter 4 runs green for a month. Then finance notices refund totals drifting downward — no crash, no traceback, no red badge. Reconstruction shows the clean refactor (the one Chapter 4 convicted) had been dropping refund_id for split-shipment children for four weeks. Chapter 5’s .get() silence would have done the same even faster: every affected row computed a total without its refund, wrote it to the report, and moved on. The program never failed. It just lied, quietly, 12,000 rows per night.

Environment Bugs

Sun, 06 Sep 2026 21:00:00 +0000

Part II — Debugging Deterministic Software

Green here, red there, same code, same data

Every contract from Chapter 8 is in place. The suite passes on the engineer’s laptop. It fails in CI — same commit hash, same fixture file, same command. The failure is the old ghost from Chapter 4:

# laptop (passes)                    # CI (fails)
$ python jobs/run_invoices.py        $ python jobs/run_invoices.py
report.csv written, 312 rows         Traceback (most recent call last):
                                       File "billing/totals.py", line 88, in summarize
                                         row["refund_id"] = order["refund_id"]
                                       KeyError: 'refund_id'

git rev-parse matches. sha256sum fixtures/incident_input.csv matches. The engineer re-reads totals.py, re-inspects state, re-runs the boundary table — all clean, because the code is clean and the data is clean. Chapters 5–8 have nothing left to convict. That is precisely the signal: when identical code plus identical data still diverges, the divergence lives underneath both.

The Notebook Is Not the Program You See

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

What changes in Part III

Parts I and II debugged a program you could re-run: same code, same data, same container, same outcome — or a substrate diff that explained the difference. The object under inspection was small and local — a value at a line, a frame’s state, a handoff between two functions.

Part III keeps the discipline and grows the object. Nothing about the loop changes — reproduce first, find the earliest divergence, move one variable, predict the result before you look, convert the fix to a contract. What changes is what the loop points at, and how hard it is to hold still. The object moves outward, one chapter at a time:

Hidden Notebook State

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

The variable that exists nowhere

Chapter 10 closed with a green, in-order Run All — gutter monotonic, cells top-to-bottom. The engineer reopens the same notebook a week later, runs the first six cells interactively, and types threshold in cell 7:

# cell 7 (fresh session, only cells 1-6 executed)
print(threshold)
# 0.9   <- where did this come from?

A search for threshold across every visible cell returns nothing. No definition, no import, no magic. Yet the kernel answers 0.9. Restart & Run All from the top fails at cell 7 with NameError: name 'threshold' is not defined — while the warm kernel insists the variable is real.

Reproducible Notebooks

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

Green here, red there, same notebook

Chapters 10 and 11 closed with a notebook that runs clean: order verified, namespace explained, Run All green. The engineer sends it to a colleague. Same file, same Run All — different number:

# author laptop (passes)                  # colleague laptop (fails)
Cell 18: val_accuracy = 0.91              Cell 18: val_accuracy = 0.84
Run All: green                            Run All: green — but different answer
# and on the author's own machine, rerun: 0.88

No traceback. No ghost variable. Three executions, three answers, zero errors. Worse: re-running on the same machine drifts too.

Debug the Data Before the Model

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

The model bug that was never in the model

The notebook reproduces (Chapter 12): pinned env, seeded runs, hashed data, green Run All on two machines. And the fraud classifier is a star — 0.99 validation AUC overnight. The engineer begins tuning the architecture for the last point of precision. Then someone deploys it on last week’s live transactions: 0.61. Coin-flip with confidence.

Shapes, Types, Devices, and Tensors

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

Same error, two diseases

The data is honest now (Chapter 13): clean split, quarantined columns, loader contracts. Training starts — and dies on the first batch:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

The engineer moves the model to CUDA. Next run, a sibling error:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x128 and 256x64)

Both are RuntimeErrors from the same training step. The team debates “a device/config problem” vs. “a layer-dimension problem” — and both sound right, because the traceback points at the matmul line in both cases while the actual handoff broke frames earlier.

When Training Goes Wrong

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

The loss curve that says nothing — three times

Tensors flow (Chapter 14): every handoff green, batches well-formed, loss computes. And the curve is dead:

epoch 1: loss=2.303  epoch 5: loss=2.302  epoch 20: loss=2.302

Flat at ln(10) ≈ 2.303 — chance-level for ten classes. The team has seen this before: “learning rate too low, crank it.” They crank it. Now:

epoch 1: loss=2.301  epoch 2: loss=nan

Three candidate explanations circulate: the optimizer step is wrong (LR/pathology), the data teaches nothing (labels shuffled or inputs normalized to constants), or nothing is actually wrong — the logged number is not the optimized number (logging averages the wrong tensor, eval runs on an unaugmented copy, the curve plots a stale buffer).

Debugging Evaluation

Sun, 06 Sep 2026 21:00:00 +0000

Part III — Debugging Interactive and Numerical AI

0.97 and broken

Training is honest now (Chapter 15): gradients flow, the overfit gate passes, the curve descends on clean data through asserted handoffs. Validation accuracy: 0.97. The demo delights. Then the engineer tries ten hand-written cases reflecting actual intent — paraphrased requests, edge phrasings, the minority class the customer cares about:

validation accuracy: 0.97 (MEASUREMENT, n=2000, seed 42)
intent slice (hand-written, n=40): 0.55 (MEASUREMENT, same weights, same code)

The eval says pass. The intent says fail. Both numbers are real measurements of the same model.

Debugging What You Cannot See

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

The glass wall

Chapter 16 ended with an honest instrument and an unanswered question. The repaired evaluation reports, repeatably: the model fails the intent slice — wrong refund answer on split shipments, confident, cited, and wrong. No leak explains it. No metric swap rescues it. No seed relitigates it.

So the engineer does the natural thing. She opens the model and looks inside — and finds there is no inside to open. No stack frame at the wrong line. No variable holding the wrong value. No refund_id key to watch go missing at a handoff. There is a weights file measured in gigabytes, an API endpoint, and a text answer. The failure is behind glass: observable only through inputs and outputs, never by inspection.

Is the Model Actually the Problem?

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

Everybody blames the weights

Chapter 17 closed with five suspects hiding behind the same glass: prompt, retrieval, context, parameters, weights. The team’s reflex is to blame the last one — “the model can’t do split shipments, we need a bigger model” — and the vendor’s reflex is to agree, helpfully, in the direction of a larger invoice.

Concrete failure. The support bot answers a split-shipment refund wrong: it cites the general 30-day policy, ignores the section 4.2 exception sitting in the knowledge base, and computes the wrong amount. The ticket says “model failure.” That label is a HYPOTHESIS, not an OBSERVATION — and in this author’s experience it is wrong about half the time, because everything the weights did was downstream of everything the pipeline handed them.

Inspect the Actual Model Input

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

What you sent is not what it got

Chapter 18 convicted the pipeline for the split-shipment refund failure. The engineer now opens her prompt file — clean, correct, citing policy 4.2 — and concludes the context was fine. It was not. What she wrote never travels alone: a system prompt prepends it, a chat template wraps it in role markers, retrieval inserts (or fails to insert) documents, and a tokenizer re-cuts every word into ids the model actually reads. Memory of the prompt is not evidence of the input.

Context Windows and Truncation

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

The case of the vanishing exception

Chapter 19 certified the rendered bytes — section 4.2 present, ids round-tripped. Yet the refund failure persists on long cases: short split-shipment queries pass, the same question with a full ticket history fails. The bytes were complete at render time and incomplete at generation time. Between the two stands a silent editor: the context window.

Concrete failure. A 14,000-token assembled context (system + six retrieved sections + two years of ticket history + the question) enters a pipeline whose effective limit is 8,000 tokens. The assembler truncates from the middle — history kept, question kept, the retrieved exception dropped without a log line. The short fixture never notices; production always does.

Sampling Is Part of the Program

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

The test that passes on Tuesdays

Chapters 19–20 certified the input: bytes complete, ledger balanced, nothing cut. And the split-shipment fixture still flickers — 9/12 Monday, 5/12 Wednesday, same bundle, same revision. The engineer re-runs the failing case once, watches it pass, and closes the ticket. Friday it reopens. Nothing changed except the draw.

Concrete failure. At temperature 0.7 the refund answer cites 4.2 in six of ten runs and the general policy in four — same bytes, same weights, four different wrong amounts shipped to four different customers. The single-run “repro” measured a sample, not the system. The debugging object was never one output; it is the distribution over outputs that this input plus these parameters define.

Internal Signals

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

The dip at the exact wrong word

Chapter 21 stabilized the distribution — deterministic params, measured pass@1, flicker gated. One residue remains: on the surviving failures, can anything inside the run locate where the answer went wrong? The engineer opens the token logprobs. At the exact token where the citation flips from 4.2 to the general policy, the logprob dips hard — the model, in some narrow sense, “hesitated” there. Tempting to declare the dip the cause. It is not. It is a lead.

Representation and Behavioral Diffs

Sun, 06 Sep 2026 21:00:00 +0000

Part IV — Debugging Models

The upgrade that fixed everything (except the refunds)

Chapter 22 closed with a repaired veer on revision A. Then the upgrade lands: revision B, announced with higher scores, migrated over a weekend. Monday the refund fixture drops from 11/12 to 7/12 — different cases fail, the citation veers at a new token, support reopens tickets the team had closed. Nothing in the pipeline changed; the weights did. Nobody ran the old behaviors against the new weights before switching, because nobody had the old behaviors pinned as an asset.

AI as Builder, Designer, Researcher, and Reviewer

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

Models are now collaborators — debug their work products

Part IV established behavior-first debugging of opaque models: swap probes that change one input property at a time, input inspection that verifies what the model actually received, and sampling discipline that refuses single-run verdicts. The model was the patient on the table.

That relationship just changed. Models are now collaborators producing code, designs, and research — work products you will merge, ship, and cite. Debug their work products, not just their outputs. An eloquent answer that fails its acceptance criteria is not a curiosity; it is a defective artifact entering your system.

Debugging Intent

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

The output is wrong — or the intent was never written

Chapter 24 routed defective work products by role. Most routes converge here: the builder built the wrong thing correctly because the intent was a sentence where a specification should have been. The debugging target shifts from the artifact to the artifact’s missing contract.

OBSERVATION: a prompt reading “add pagination to the user list” returns working code that the requester rejects — wrong page size, wrong sort, wrong error shape. The code runs; the intent was never specified. HYPOTHESIS H1 (underspecified intent): the prompt admits ≥3 compliant implementations, and the model picked one the requester did not want. H2 (model capability failure): no reasonable reading of the prompt produces the rejected behavior. INFERENCE: none yet — H1 and H2 are separable only by an intent artifact written before regeneration. Blaming the model before pinning intent is single-run inference.

Debugging Context for Coding Agents

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

The contract was clear — the agent never saw the files

Chapter 25 pinned intent. Now the agent with a perfect contract edits the wrong module, reimplements an existing helper, and misses the migration the contract names by path. The intent artifact is innocent. The working set is the suspect.

OBSERVATION: the agent’s session log shows 4 files read; the fix required 7, including the migration and the helper the agent duplicated. HYPOTHESIS H1 (missing-file): the agent never received the decisive file — retrieval omitted it or the working set excluded it. H2 (misread-file): the file was in context but the agent’s edits contradict its content — present but unused. H3 (stale context): the file was read but an older revision — edits fit a version that no longer exists. INFERENCE: none yet — H1/H2/H3 predict different working-set dumps and are separable only by one.

Debugging AI-Generated Designs

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

The diagram is beautiful — and over budget by 250 milliseconds

Chapter 26 verified the agent saw the files. Now it proposes a profile endpoint fanning out to three services per request, with a sequence diagram reviewers call “clean.” The p99 budget is 200 ms. The line items sum to 450. Eloquence is not a load test.

OBSERVATION: the design document claims “meets latency goals” while listing per-hop latencies (auth 80 ms, profile 190 ms, recommendations 180 ms) that sum past the stated 200 ms budget before queueing. HYPOTHESIS H1 (constraint violation): the design contradicts its own numbers — no implementation of this shape can meet budget. H2 (constraint underspecification): the budget was never pinned, so “meets goals” is unverifiable prose rather than a checkable claim. H3 (tradeoff blindness): a viable alternative (cached reads, collapsed hops) exists but was never generated or compared. INFERENCE: none yet — H1/H2/H3 predict different rows in a constraint table and are separable only by one.

Debugging AI Research

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

The literature review reads perfectly — and one paper doesn’t exist

Chapter 27 priced the design. Now its “related work” section summarizes three papers with DOIs, quotes, and confident synthesis — one citation resolves to an unrelated paper, one has no retrievable byte anywhere, and the third says the opposite of what is claimed. The prose is flawless. The provenance is fiction.

Debugging Coding Agents

Sun, 06 Sep 2026 21:00:00 +0000

Part V — Debugging AI-Assisted Development and Research

The agent worked for 40 minutes — and the suite is still red

Chapters 24–28 debugged the agent’s work products, intent, context, designs, and sources. Now the trajectory itself is the patient: a code agent loops plan → edit → test for 40 minutes, re-applies the same patch three times, re-runs the same failing test, and closes with “fixed and verified” on a red suite. The final diff looks almost right. The path getting there is the defect.

Treat Prompts as Programs

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

The prompt that “just needed a tweak” — and took down the refund bot

Chapter 29 ended with the trajectory triaged and the loop convicted: the agent re-applies equivalent edits because nothing in its instructions forbids equivalent retries, requires exit-code gating, or defines progress. That “nothing” has a name. It is the prompt — a program with no repository, no version, no diff, and no test, edited by folklore until it breaks in production.

Minimize the Prompt

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

The 40-line prompt that fails — and nobody knows which line

Chapter 30 versioned the prompt. Now the version is long: fourteen months of appended instructions, three example blocks, two guardrail paragraphs, a formatting schema, and a “be concise but thorough” contradiction nobody remembers adding. It fails on refund reference RB-8814 — invents a number — and every proposed fix appends line 41.

Retrieval Is a Pipeline

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

The ledger chunk exists — and the answer says it does not

Chapters 30–31 versioned and minimized the prompt. Now the minimal prompt still fails: asked about refund RB-8814, the assistant invents a “processed” status while the ledger line “refund PENDING, no reference issued” sits in the corpus. The prompt is exonerated. The words arrived; the evidence did not.

Concrete failure. The retriever returns five chunks for “refund RB-8814 status.” The ledger chunk is not among them. The engineer re-embeds everything, the failure persists, and the team debates the embedding model — while the actual break sits one stage downstream, invisible because “retrieval” is treated as one box.

Retriever Failure or Generator Failure?

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

Same wrong refund answer, three different culprits

Chapter 3 previewed this triple in miniature; Chapter 32 staged the pipeline. Now the full attribution runs. The symptom is fixed: “Your refund was processed on September 2; reference RB-8814.” The corpus truth is fixed: ledger line “refund PENDING, no reference issued,” no RB-8814 anywhere. Three incidents in three weeks produced this identical sentence — and each had a different first divergence.

Debugging Hallucinations

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

The answer is half right — which half do you act on?

Chapter 33 convicted the generation boundary for incident C. Now the harder case arrives: an answer with four claims, three grounded and one invented, delivered in one fluent paragraph. “Your refund is pending (ledger confirms), no reference has been issued (ledger confirms), processing typically takes 3–5 days (policy doc confirms), and reference RB-8814 was assigned September 2 (nothing confirms).” Finance can act on the first three. The fourth, acted on, emails a customer a fiction.

The Model's Explanation Is Not a Trace

Sun, 06 Sep 2026 21:00:00 +0000

Part VI — Debugging Prompts, Retrieval, and Hallucinations

“I retrieved the ledger, confirmed RB-8814, and answered” — none of which happened

Chapter 3 stated the rule; Chapters 30–34 built the machinery that makes it enforceable. Now the capstone case. After the invented RB-8814 answer, the engineer asks the model to explain itself. It replies with a crisp chain-of-thought: “First I retrieved ticket #4471 and the refund ledger; both confirmed RB-8814; therefore I answered processed.” Every step is fluent. The retrieval log shows no ledger chunk. Ticket #4471 is another customer’s. The explanation is a second fabrication about the first fabrication — and it arrives shaped exactly like a trace.

An Agent Is a Trajectory

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

Single generations are over — the debugging object is now the trajectory

Part VI treated the prompt as a program and pinned every claim to either retrieval or generation. That discipline holds for one call. It breaks the moment the system makes five of them: a support agent looks up order 8841, checks the refund policy, calls issue_refund, re-checks the balance, and apologizes — twice refunding the customer while its final message reads perfectly. The generation is clean. The trajectory is the defect.

Trace the Agent

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

The log says “tool ran” — and nothing else

Chapter 36 promoted the failure to a trajectory. Now the practitioner opens the log and finds this: step 5: search_orders — ok, step 6: issue_refund — ok, step 9: done. No arguments, no returns, no state hashes, no latencies, no truncation flags. The double-refund from Chapter 36 is back, and the trace cannot separate any of its three hypotheses — the observation that would convict one of them was never written down.

Agent Failure Taxonomy

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

Same wrong answer, four different diseases

Chapters 36–37 made the trajectory readable. Now the practitioner faces the next trap: two refund-agent failures look identical — “refunded twice” — but the contracted traces convict different steps. Run A repeats issue_refund because the plan listed two refunds before any tool ran. Run B plans one refund but re-issues after a timeout it never saw. Identical ledgers, opposite repairs: fixing the plan cures A and leaves B broken; adding timeout handling cures B and leaves A double-spending.

Loops, Thrashing, and Retry Storms

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

The agent worked for 90 steps — and the state never moved

Chapter 38 classifies single divergences. Now the practitioner watches a research agent burn 90 steps and a full API budget: search_papers → fetch_pdf → parse_failed → search_papers (same query) → fetch_pdf (same URL) → parse_failed … The contracted trace is complete, every step classifies somewhere, and the run still fails — because the defect is not one wrong step but zero net movement across thirty of them.

Time Travel, Replay, and Forking

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

“What if it had branched at step 12?” — now answerable

Chapter 39 halts loops without explaining them. The practitioner is left with the obvious unasked question: at step 12 the agent re-issued the failed fetch_pdf; had it instead switched sources, would the run have succeeded? Re-running the whole task from scratch cannot answer that — a fresh run diverges everywhere, and any success is attributable to nothing. The counterfactual needs machinery: restore the exact state at step 12, intervene once, and run forward.

Causal Replay

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

The fork worked — so the forked step caused it? Not yet

Chapter 40’s fork F1 (HTML instead of PDF at step 12) succeeded in 3/3 trials. The practitioner is tempted to close the case: the PDF action caused the failure. But the forward runs also used a different source snapshot, ran at a different hour, and consumed a parse path with its own cache — any of which could carry the change. Sensitivity shows that the downstream moved; it does not show what moved it.

Trajectory Diff

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

Two runs, same task, opposite ledgers — diff the time, not the text

Chapters 40–41 test one run’s counterfactuals. Now the practitioner holds something more common: run P (passing — one refund, ledger clean) and run F (failing — two refunds) on the same task, same pins, same day. The final answers differ by one sentence. The tool logs differ everywhere — retries, latencies, paraphrases — and naive text-diffing buries the one step that matters under forty cosmetic ones.

Multi-Agent Systems

Sun, 06 Sep 2026 21:00:00 +0000

Part VII — Debugging Agents

Three agents, one failure, zero owners

Chapters 36–42 debugged one agent’s trajectory. Now the practitioner inherits a support pipeline with three: a triage agent classifies the ticket, a researcher agent gathers policy evidence, a action agent issues the refund. The ledger shows the familiar double-refund — and each agent’s local trace looks clean. Triage passed the right category; research cited the right policy; action executed what it received. The defect lives in none of the trajectories and in all of the gaps between them.

Can One AI Debug Another?

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The assistant that explained everything and proved nothing

Chapter 43 closed with a direct question: the trajectories are routable by human practitioners — but can any of that discipline be delegated to an AI system without inheriting the failure modes this book forbids? Here is the concrete version. A practitioner hands a failing agent trajectory to a capable assistant model: “find the bug.” Minutes later the assistant returns a fluent, confident report — root cause identified, three supporting quotes, a suggested fix. The practitioner applies the fix. The symptom disappears once, returns the next day, and nobody can say which of the assistant’s claims was ever evidence and which was narration.

The AI Crash Dump

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The failure that could not be reopened

Chapter 44 permitted the AI assistant to propose — but only from a frozen, hashed bundle. That bundle did not yet exist as a defined artifact. Here is why it must. A practitioner investigates a support chatbot that quoted a refund policy nobody can find. She screenshots the answer, copies the prompt from memory, and asks the team to reproduce it. Three days later nobody can: the model version has rolled forward, the retrieved documents have been re-indexed, the temperature was never recorded, and the “prompt” she saved differs by two sentences from what was actually sent. The failure is real, witnessed — and undebuggable.

Diagnostic AI Invariants

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The assistant that broke every rule politely

Chapter 45 froze the past; this chapter polices the consumer. A practitioner runs a well-bundled failure through a debugging assistant that returns a report violating, in one page, every rule Chapters 01–03 established: its “observations” are paraphrases, its single hypothesis arrives with no competitor, its conclusion cites the assistant’s own confidence and one passing run, and the fix ships to production without human review. The bundle was perfect. The discipline was absent — because nothing checked it.

From Symptom to Hypotheses

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The clever hypothesis that missed the boring cause

Chapter 46 can reject single-story reports — but it needs a defined space of competitors to judge plurality against. Here is the failure that space prevents. A RAG answer fabricates a citation. The assistant proposes one elegant hypothesis — “the reranker demoted the gold chunk” — with a sophisticated test involving score analysis. The team spends a week on reranker tuning. The actual cause is mundane: a stale index snapshot meant the gold chunk was never retrieved at all. Nobody listed it because nobody enumerated; cleverness covered one branch while coverage would have covered all of them.

Discriminating Experiments

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

Three hypotheses, nine runs, one lesson

Chapter 47 mapped the space; this chapter prices the search. A practitioner holds three live hypotheses for a RAG fabrication — stale snapshot (H1), reranker demotion (H2), prompt override (H3) — and runs them naively: re-index (expensive, hours), retrain the reranker (days), rewrite the prompt five ways (scattershot). Nine runs later H1 is confirmed by accident — the re-index fixed it — but the prompt rewrites also “helped” once each, the reranker retrain changed scores without changing outcomes, and nobody can say which run proved what because two variables moved per run and predictions were written after seeing outputs.

How Do You Know the Diagnosis Is Right?

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The diagnosis that survived everything except verification

Chapter 48 split the space to one survivor — but survival is not proof. A practitioner holds H1 (stale snapshot) as the last branch standing for the fabricated citation: the chunk was absent, expansion changed nothing, wording swaps changed nothing. She ships the re-index, the symptom disappears for a week, and the postmortem records “root cause: stale index.” A month later the fabrication returns with a fresh index. The original evidence never distinguished “stale snapshot caused it” from “stale snapshot accompanied it” — no prediction-match ledger, no independent verification evidence, no human sign-off on the causal claim. The branch survived; the diagnosis was never verified.

AIDebugBench

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The leaderboard that measured nothing

Chapter 49 verified single diagnoses; this chapter asks what would measure debugging skill itself. A team ships “DebugScore”: an assistant is given fifty broken AI setups, its fixes are graded by whether the symptom disappears, and the resulting number — 87% — is cited in planning docs as proof the debugger works. Nobody asks whether the tasks resemble the book’s failure modes, whether grading checked exoneration of alternatives or just relief, whether the assistant had seen the tasks in training, or whether one trial per task decided each score. The number is precise, reported, and evidentially empty.

Debug the Debugger

Sun, 06 Sep 2026 21:00:00 +0000

Part VIII — Building the AI Debugger

The checker that passed everything

Chapters 44–50 built an impressive machine: gated assistants, frozen bundles, invariant checkers, enumerated spaces, discriminating runs, verification ledgers, a benchmark design. Then a practitioner notices the invariant checker has passed every report for a month — including one she hand-verified as containing an unresolving citation. The cause is embarrassing: a bundle migration changed the hash algorithm, the checker’s resolver silently failed open, and every I-2 verdict since is theater. The debugging system debugged everything except itself.

AI Observability

Sun, 06 Sep 2026 21:00:00 +0000

Part IX — Production Debugging and Prevention

The refund that nobody can replay

Part VIII built machine-assisted debugging machinery: contracted traces, hypothesis generation under evidence rules, discriminating experiments checked by intervention outcomes. That machinery means nothing without production observability. A practitioner inherits exactly this: a support agent double-refunds one customer on Tuesday, the on-call engineer asks what the model saw, and the answer is a dashboard screenshot of the output text plus a log line reading POST /chat 200 812ms. No prompt hash, no retrieved documents, no parameters, no seed. The incident is real, the evidence is gone, and every diagnosis from here on is storytelling.

From Production Failure to Regression

Sun, 06 Sep 2026 21:00:00 +0000

Part IX — Production Debugging and Prevention

The fix that fixed nothing — twice

Chapter 52 ended with a replayable record: the double-refund request re-assembled bit-exact, reproduced 2-of-3 trials, guardrail verdict empty. The on-call engineer ships the obvious repair — a prompt sentence forbidding double authorization — and closes the incident. Six weeks later the same double-refund returns under a reworded ticket, and nobody can say whether the old fix failed or a new defect arrived: no bundle was frozen, no repro was pinned, no regression test exists. The team fixed forward twice and learned nothing once.

Runtime Invariants and Guardrails

Sun, 06 Sep 2026 21:00:00 +0000

Part IX — Production Debugging and Prevention

The guardrail that watched the wrong door

Chapter 53 ended with a contractual repair and a bidirectional test: scope-less tickets now reject-and-ask in CI. In production, the next scope-less ticket still double-spends. The postmortem reveals why: the guardrail checked citation format at the output, while the defect lived at the handoff between research and action — an unchecked edge the guardrail never evaluated. The team owned a passing test and an unguarded path. Validation without enforcement is a wish.

Debugging Cost and Latency

Sun, 06 Sep 2026 21:00:00 +0000

Part IX — Production Debugging and Prevention

The invoice nobody can attribute

Chapter 54 ended with enforced checkpoints: known-bad acts refuse before they commit. Then finance forwards a $40,000 monthly inference bill — triple the forecast — alongside p99 latency at 90 seconds, and the team argues from vibes. “The model got expensive” blames the vendor; “users ask longer questions” blames the customers; “add caching” prescribes before attributing. The Chapter 52 records hold per-stage segments, but nobody has ever assembled them into a ledger. Spend and delay are debuggable quantities. The team is debugging them as weather.

Debugging in Production

Sun, 06 Sep 2026 21:00:00 +0000

Part IX — Production Debugging and Prevention

02:14, three responders, zero procedure

Chapter 55 ended with an attributed ledger and joint-verdict optimizations — peacetime instruments. At 02:14 the refund agent starts authorizing duplicates at volume, the page fires, and three responders improvise three different incidents: one restarts the service (destroying in-flight evidence), one edits the prompt in the dashboard (a multi-variable repair under fire), one announces “we’re investigating” in the status channel and then goes silent for forty minutes. By dawn the bleeding stopped — nobody knows which act stopped it, the bundle was never frozen, and the postmortem reconstructs courage instead of causation.

The Ten-Minute Debug

Sun, 06 Sep 2026 21:00:00 +0000

Part X — The Debugging AI Playbook

Ten minutes, one failing request, no permission to boil the ocean

Part IX closed with the full method under production load: emit, convey, enforce, book, respond in order. Now the practitioner gets reality: ten minutes before the deploy review, one user-flagged refusal that “worked yesterday,” and no authority to freeze slices or convene commanders. The temptation is full rigor in miniature — a five-stage conveyor, a calibration set, and a ledger before lunch. That fiction produces the worst outcome: ten minutes spent starting everything and finishing nothing, with a confident verdict announced to fill the silence.

The One-Hour Investigation

Sun, 06 Sep 2026 21:00:00 +0000

Part X — The Debugging AI Playbook

One hour, one defect, pins available

Chapter 57 ended with a routed handoff: the flagged refusal triaged, pins UNKNOWN, no verdict, hour queue. Now the practitioner sits down with sixty protected minutes, log access, pinned environment, and the handoff record. The failure is concrete: the research agent’s answers degraded after Tuesday’s index refresh — some queries cite stale policy, others refuse outright. One hour is the book’s standard depth: enough to isolate, not enough to certify.

The Full AI Incident Investigation

Sun, 06 Sep 2026 21:00:00 +0000

Part X — The Debugging AI Playbook

The incident that owes a published record

Chapter 58 ended with isolation and queued artifacts: one defect, one repair candidate, deeper questions deferred. Now the incident that defeats the hour: a weekend of duplicate refunds across hundreds of accounts, two services disagreeing on whose handoff dropped scope, a regulator asking what happened, and leadership asking what prevents recurrence. Triage routed it; the hour isolated one edge; neither suffices. This incident owes customers remediation, the team prevention, and the record an account — with named roles producing named artifacts on a visible schedule.

The Debugging AI Toolkit

Sun, 06 Sep 2026 21:00:00 +0000

Part X — The Debugging AI Playbook

Sixty chapters, one workbench — and the question of what it proved

Chapter 59 ended with the published packet: one incident fully accounted, generality bounded on the final page. Now the practitioner faces the opposite problem — not one incident with a full procedure but any symptom with sixty chapters behind it. Which instrument, in which order, for this failure? The failure is meta and concrete at once: a teammate asks “the agent is wrong and slow — where do I start?” and the answer cannot be “read the book again.” Sixty chapters that cannot be selected under pressure are sixty chapters shelved.

Debugging AI on Programmer.ie: Modern AI programming

What Does It Mean to Debug?

The puzzle: two fixes, one bug

The First Divergence

The puzzle: the report is wrong, but where?

Evidence Before Explanation

The puzzle: the model explains, fluently, and is wrong

The Debugging Stack

The puzzle: three suspects, one KeyError

Reading Python Exceptions

The crash that names the wrong culprit

Inspect State, Don't Guess

The function that reads correctly and runs wrong

Debug the Boundary

The suite is green and the refunds are wrong

Assertions, Invariants, and Contracts

The corruption nobody heard

Environment Bugs

Green here, red there, same code, same data

The Notebook Is Not the Program You See

What changes in Part III

Hidden Notebook State

The variable that exists nowhere

Reproducible Notebooks

Green here, red there, same notebook

Debug the Data Before the Model

The model bug that was never in the model

Shapes, Types, Devices, and Tensors

Same error, two diseases

When Training Goes Wrong

The loss curve that says nothing — three times

Debugging Evaluation

0.97 and broken

Debugging What You Cannot See

The glass wall

Is the Model Actually the Problem?

Everybody blames the weights

Inspect the Actual Model Input

What you sent is not what it got

Context Windows and Truncation

The case of the vanishing exception

Sampling Is Part of the Program

The test that passes on Tuesdays

Internal Signals

The dip at the exact wrong word

Representation and Behavioral Diffs

The upgrade that fixed everything (except the refunds)

AI as Builder, Designer, Researcher, and Reviewer

Models are now collaborators — debug their work products

Debugging Intent

The output is wrong — or the intent was never written

Debugging Context for Coding Agents

The contract was clear — the agent never saw the files

Debugging AI-Generated Designs

The diagram is beautiful — and over budget by 250 milliseconds

Debugging AI Research

The literature review reads perfectly — and one paper doesn’t exist

Debugging Coding Agents

The agent worked for 40 minutes — and the suite is still red

Treat Prompts as Programs

The prompt that “just needed a tweak” — and took down the refund bot

Minimize the Prompt

The 40-line prompt that fails — and nobody knows which line

Retrieval Is a Pipeline

The ledger chunk exists — and the answer says it does not

Retriever Failure or Generator Failure?

Same wrong refund answer, three different culprits

Debugging Hallucinations

The answer is half right — which half do you act on?

The Model's Explanation Is Not a Trace

“I retrieved the ledger, confirmed RB-8814, and answered” — none of which happened

An Agent Is a Trajectory

Single generations are over — the debugging object is now the trajectory

Trace the Agent

The log says “tool ran” — and nothing else

Agent Failure Taxonomy

Same wrong answer, four different diseases

Loops, Thrashing, and Retry Storms

The agent worked for 90 steps — and the state never moved

Time Travel, Replay, and Forking

The puzzle: three suspects, one `KeyError`