Part IV — Debugging Models

The dip at the exact wrong word

Chapter 21 stabilized the distribution — deterministic params, measured pass@1, flicker gated. One residue remains: on the surviving failures, can anything inside the run locate where the answer went wrong? The engineer opens the token logprobs. At the exact token where the citation flips from 4.2 to the general policy, the logprob dips hard — the model, in some narrow sense, “hesitated” there. Tempting to declare the dip the cause. It is not. It is a lead.

Concrete failure. The refund answer begins correctly (“Under section 4…”) then veers: token “general” arrives with logprob −6.2 against a running mean near −0.4, entropy spiking across the alternatives (4.2-exception, 4.1, general-policy all live). The attention map, pulled up next, lights over the general-policy paragraph. Two weak signals, both pointing near the veer — and neither allowed to convict anything on its own.

OBSERVATION: logprob dip (−6.2 vs. −0.4 mean) at the veer token (MEASUREMENT, deterministic params, pinned revision); attention mass concentrates on the general-policy segment at that step. HYPOTHESIS H1 (genuine uncertainty): the dip marks competing attractors — the run is truly undecided between 4.2 and general, and a boundary probe will flip it. HYPOTHESIS H2 (signal mirage): the dip is a surface artifact (rare token, formatting boundary, tokenizer seam) — boundary probes ignore it and the answer holds regardless. HYPOTHESIS H3 (misread attention): the attention highlight is illustrative overlap, not mechanism — moving the highlighted text changes nothing while moving unhighlighted text flips the answer. INFERENCE: none yet — signals locate candidates; only boundary interventions promote a candidate to a cause.

This chapter’s question: how do you use interior signals as triage leads — locating where to probe — without ever letting them testify as explanations?

Why “the heatmap shows why” fails first

The obvious move — screenshotting attention (or the logprob dip) as the root cause — fails because these signals correlate with behavior without access to its causation. Four mirage traps:

1. Attention-as-explanation. Attention weights describe one routing step among many, not the decision. They illustrate where mass fell on this run; they do not show what would happen if the mass moved. Cause requires the counterfactual — run the moved version, or UNKNOWN.
2. Confidence-as-truth. High logprob (or a volunteered “I’m 95% sure”) read as correctness. Logprobs measure next-token predictability under training statistics, not truth against the world. Fluent falsehoods are high-confidence by construction.
3. Dip-as-diagnosis. A low-probability token treated as the defect. The dip marks where the run surprised its own statistics — sometimes genuine indecision, sometimes a rare-but-correct name, date, or id. The signal cannot tell those apart; only a probe can.
4. Correlation-as-cause. “Failures show high entropy, so entropy causes failure.” Both may follow a third fact (ambiguous input, truncated context). Steering the signal (suppressing uncertain tokens) then hides the thermometer instead of cooling the room.
5. Cherry-picked signal. Reporting the one run where attention agreed with the theory and omitting the nine where it pointed elsewhere. Signals auditioned across runs and reported once are decoration, not evidence.

OPINION: interior signals are the model’s body language — worth glancing at to decide where to look next, worthless as testimony. The debugger who convicts on body language convicts on demeanor. Glance, nominate, probe — in that order, every time.

The mental model: weak signals for triage, never verdicts. Logprob dips and entropy spikes locate (which token to probe first); attention illustrates (which segment to move first); uncertainty routes (review queue vs. auto-ship). Each signal earns its keep only when a boundary intervention it suggested flips the outcome it predicted. Until then it is a pointer with no authority.

The method: locate with signals, convict with probes

1. Capture signals alongside the bundle. Per-token logprobs (where the reader’s endpoint exposes them — availability is per-provider and changeable; unsourced absence is UNKNOWN, not zero), entropy or top-k spread at the veer, attention snapshot if cheaply available. All hashed with the run; all labeled with revision.
2. Nominate, don’t convict. The dip nominates the veer token; attention nominates a segment. Each nomination becomes a boundary probe with a FORECAST: “forcing the 4.2 citation at the veer flips the completion to correct ≥8/10” or “removing the highlighted general-policy paragraph (nothing else) flips the answer.”
3. Run the probes. One intervention per run, ≥5 trials, deterministic params. Prediction if H1: the nominated probe flips the outcome as forecast. Prediction if H2/H3: the nominated probe moves nothing while an un-nominated intervention flips it — the signal pointed at the crowd, not the culprit.
4. Route by uncertainty, decide by probe. Whatever the signals said, shipping decisions follow probe outcomes: high-entropy or dip-marked runs queue for review (triage use — legitimate), but no run is declared wrong or right from its signals (verdict use — forbidden).

    flowchart TD
    S["capture signals with the bundle: per-token logprobs, entropy / top-k spread, attention snapshot"] --> N["nominate in writing: veer token (logprob dip) + attended segment — candidates, not causes"]
    N --> A["probe-A: intervene exactly where the signal pointed (forced prefix / segment removal), >=5 trials"]
    N --> B["probe-B control: an un-nominated intervention, >=5 trials"]
    A --> V{"outcome pattern?"}
    B --> V
    V -->|"A flips >=8/10, B flat"| H1["H1: genuine uncertainty — veer located, not explained; route to input repair"]
    V -->|"A flat, B flips"| H2["H2/H3: signal mirage — retire the signal for this fixture"]
    V -->|"neither flips as forecast"| U["UNKNOWN — name the next probe"]
    H1 --> R["route by uncertainty for a review queue only — never a correctness label"]
  

A note on which uncertainty signal to route with. The single-run token logprob dip is the weakest usable form. The signals with measured detection value are distributional: Farquhar and colleagues compute semantic entropy — entropy over clusters of same-meaning answers across many samples — and show it detects confabulations across tasks with no task-specific tuning (Farquhar et al., 2024); Manakul and colleagues’ SelfCheckGPT needs no logprobs at all, sampling several responses and flagging the claims that disagree between them (Manakul, Liusie & Gales, 2023). Both build directly on Chapter 21’s trial series. Route with a cross-sample signal where you can afford the samples; fall back to the single-run dip only when you cannot.

# signals locate; probes convict (deterministic params, pinned revision)
run = generate(bundle, output_signals=["logprobs", "topk", "attention"])
veer = argmin_token_logprob(run)          # MEASUREMENT: dip location + depth
print("veer:", veer.token, veer.logprob, "entropy:", veer.entropy)
print("attended:", top_attention_segment(run, at=veer))  # illustration only
# H1 probe: intervene exactly where the signal pointed, one variable, >=5 trials
print("force-citation:", run(bundle.force_prefix("section 4.2-exception"), trials=5))
print("remove-highlight:", run(bundle.without_segment("general-policy"), trials=5))
# FORECAST: H1: a signal-nominated probe flips >=8/10; H2/H3: nominated probes flat,
# un-nominated intervention flips -- signal demoted to coincidence.

OBSERVATION (constructed illustration, not a measured run): dip −6.2 at “general” with three live alternatives; force-citation probe 9/10 correct completions; remove-highlight probe 8/10; reorder-unhighlighted-history probe 2/10. UPDATED BELIEF: H1 supported at the veer — the dip marked genuine competition between two held answers, and both nominated probes move it. Attention’s highlight agreed here but earns no general credit: agreement on one run is not validation of the signal. INFERENCE: the veer is located, not explained — the cause remains the input competition (Chapters 19–20 jurisdiction), now precisely addressed; entropy-threshold review queuing is justified as triage, never as correctness labeling.

Note the control’s role: probe-B (the un-nominated intervention) is what keeps probe-A honest. Without it, any flip after the nomination looks like confirmation — the classic correlation trap with a heatmap garnish. With it, the chapter’s hard rule is enforceable on paper: a signal without a winning nominated probe and a losing control is a coincidence with formatting.

Research lineage: present is not used, and one run is not a signal

Probing shows information is present, not that the model uses it. Belinkov’s survey of probing classifiers makes the methodological point this chapter is built on: a probe that reads a feature out of the activations at high accuracy tells you the feature is encoded, not that it drives the output — establishing the latter needs a causal intervention that removes or alters the feature and measures the behavior (Belinkov, 2022). “Locate with signals, convict with probes” is that principle stated for practitioners; the interior-signal equivalent of the chapter’s boundary probe is activation patching, where a research setup literally swaps a hidden state and checks whether the answer follows.

The reliable uncertainty signal is distributional. Semantic entropy (Farquhar et al.) and self-consistency checking (Manakul et al., SelfCheckGPT) both work by comparing multiple samples rather than reading one run’s logprobs, and both beat single-sequence likelihood as confabulation detectors. This is the same lesson as Chapter 21: the object worth measuring is the distribution, not the draw — and that holds for interior signals too. Be precise about what “detector” buys, though. Three capabilities pull apart: detect (this response is probably a confabulation), localize (this token or span is the wrong one), explain (this is why the weights preferred it). A multi-sample meaning-cluster signal reaches the first on QA-style tasks and no further — it does not point at the veer token, it does not name a mechanism, and it fails silently on confident fabrication, where every sample lands on the same wrong answer so the cluster entropy stays low. That is the exact sense in which interior signals never convict: good detectors exist, localizers and explainers do not, and even the good detectors have a blind spot a probe would catch. The cheap single-pass version is a hidden-state probe: Kossen and colleagues’ semantic entropy probes read a binarized semantic-entropy estimate straight off one generation’s activations, cutting the multi-sample cost to almost nothing and generalizing better out of distribution than a plain accuracy probe (Kossen et al., 2024). It is still governed by everything above — a probe that reads uncertainty out of the activations shows the uncertainty is encoded, not that it drove the wrong token (Belinkov again), and it detects without localizing.

Attention and confidence carry the caveats from Chapter 3. Attention weights can be altered without changing predictions (Jain & Wallace); saliency methods can survive model randomization (Adebayo et al.); verbalized and token-level confidence are shaped by RLHF toward decisiveness (Tian et al.). Nothing in Part IV lifts those limits — it only adds that a signal may still be a useful triage lead once a nominated probe confirms it.

Lab 22: signal-nominated probe with a pre-written flip

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own signal-plus-probe table.

Setup. Take one deterministic-params failure with signals available on the reader’s endpoint (or inject competition: include two rival policy sections in context). Freeze the bundle with signal payloads. Fix the veer definition before looking (lowest-logprob content token, or first token diverging from the reference).

Task.

1. Write H1/H2/H3 with distinct flip FORECASTs before probing (e.g., “H1: signal-nominated intervention flips ≥8/10; H2: nominated flat ≤4/10 AND un-nominated flips ≥8/10; H3: moving highlighted text ≤4/10 AND moving unhighlighted rival ≥8/10”).
2. Independent variable per run: the single signal-nominated intervention (forced prefix, segment removal, reorder). Controlled variables: revision, params, seed, everything else.
3. Record OBSERVATION (dip depth, entropy, attended segment, per-trial probe outcomes) and UPDATED BELIEF. Any verdict sentence citing the heatmap or the dip without a probe row is forbidden — mark it UNKNOWN and name the missing probe.
4. Specify the triage routing on paper: which signal thresholds queue review, with the explicit note that thresholds route labor, never declare truth.

Success criterion. A completed table where every signal sentence links to a probe row, matching one pre-written pattern, plus the written routing rule. A heatmap screenshot without probe rows is explicitly not completion.

Companion tool: Entropy / Logprob Viewer

What it accepts: the frozen bundle with per-token signals (logprobs, entropy/spread, attention snapshot where available), the nominated veer + segment, and the probe series with flip FORECASTs. What it performs: it renders the dip/entropy/attention aligned to tokens, links each nominated site to its probe, checks flip outcomes against FORECASTs, refuses any causal sentence while its probe row is missing or single-trial, and stamps signal hashes with the revision. What it can establish: where doubt concentrated on this run and whether the nominated intervention flips the outcome — triage location plus probe result, nothing more. What it cannot establish: why the weights prefer one attractor, whether attention explains anything, whether confidence tracks truth, or cross-run/cross-revision generality. It never converts a dip, a heatmap, confidence, or entropy into a cause, a correctness label, or a capability claim — single-run signals are UNKNOWN for all of those. How its output changes your next action: a probe-confirmed veer routes to input repair at the nominated segment (Chapters 19–20) with surgical precision; a mirage result (H2/H3) retires the signal for this fixture and routes to un-nominated interventions; persistent competition routes to Chapter 23 — the behavior, not the signal, becomes the regression asset.

Paper form, sufficient for this chapter:

VEER: tok ___ (lp ___ vs mean ___) | ENT ___ | ATTEND ___ (illustration only)
PROBE-A (nominated ×5): ___   PROBE-B (control ×5): ___   CONVICTION: H1 / H2 / H3 / UNKNOWN
ROUTING: queue review when ___ (triage only; never a correctness label)

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The locate-then-probe discipline precedes any automation.

Reusable procedure: every dip gets this, never more

1. Capture signals with the run — logprobs, spread, attention; revision-labeled or UNKNOWN.
2. Nominate in writing — veer token + segment, before intervening.
3. Probe the nomination — one intervention, ≥5 trials, FORECAST first.
4. Run the control — an un-nominated intervention, or the signal is untested.
5. Route, don’t rule — thresholds queue review; probes decide causes.

Failure modes

• Heatmap-as-root-cause. Shipping the screenshot as the diagnosis. Illustration is not intervention — no probe, no cause.
• Confidence-as-correctness. Auto-shipping high-logprob runs, auto-rejecting dips. Predictability is not truth; the routing threshold is a labor policy, not an oracle.
• Signal steering. Penalizing uncertain tokens to flatten the thermometer. The competition persists; only its display dies.
• Single-run signal lore. “Dips always mean X” from one incident. Signals are per-run, per-revision observations; every generalization is a fresh HYPOTHESIS awaiting its own probe series.
• Provider-fact hardcoding. Asserting signal availability, meaning, or stability across endpoints from this book. Endpoints are changeable — record what the reader’s endpoint exposes today, attribute, re-measure.
• Threshold-as-label. Promoting a review-queue cutoff into a correctness stamp (“above 0.8 means right”). The threshold allocates labor; the probe allocates belief — swapping them corrupts both.
• Signal hoarding. Collecting logprobs on every run with no nomination or probe attached. Unread signals are storage costs with confidence intervals.

Limits, per contract: one signal-plus-probe record locates one veer under one revision and param set; it explains nothing about the weights, certifies no threshold as truth, and expires on revision or param change. UNKNOWN wherever signals are unavailable, probes ran single, or the control is missing.

References

• Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 48(1), 2022, pp. 207–219. https://doi.org/10.1162/coli_a_00422
• Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting Hallucinations in Large Language Models Using Semantic Entropy. Nature 630, 2024, pp. 625–630. https://doi.org/10.1038/s41586-024-07421-0
• Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. arXiv:2406.15927, 2024. https://arxiv.org/abs/2406.15927
• Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. Proceedings of EMNLP, 2023, pp. 9004–9017. https://doi.org/10.18653/v1/2023.emnlp-main.557
• Sarthak Jain and Byron C. Wallace. Attention is not Explanation. Proceedings of NAACL-HLT, 2019. https://doi.org/10.18653/v1/N19-1357

Debugging Checklist

• Signals captured with the run (logprobs/spread/attention) + revision, or absence marked UNKNOWN?
• Veer + nominated segment written before any probe?
• H1/H2/H3 flip FORECASTs pre-written (nominated vs. control)?
• Nominated probe run ≥5 trials; un-nominated control run?
• No causal sentence cites a signal without its probe row?
• Routing rule written as triage-only (queues labor, never labels truth)?
• Uncertainty routing uses a cross-sample signal (semantic entropy / consistency) where samples are affordable, not just a single-run dip?
• Un-nominated control probe run alongside every nominated probe?
• Signal unavailability marked UNKNOWN (never read as zero)?
• No confidence/entropy reading recorded as correctness evidence?

What This Chapter Established

• Interior signals as triage-only: logprob dips and entropy locate candidates, attention illustrates, uncertainty routes review — conviction comes solely from signal-nominated boundary probes, demonstrated on the citation veer as H1 — constructed illustration, no measured runs claimed.
• Lab 22 as a proposed signal-plus-probe record the reader executes; the Entropy / Logprob Viewer contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any mechanism claim, any attention-explains behavior, any confidence-truth link, any threshold validity beyond the reader’s own probe series.
• Research grounding: probing accuracy shows a feature is encoded, not used — causal intervention is required (Belinkov); the uncertainty signals with measured detection value are distributional (semantic entropy — Farquhar et al.; SelfCheckGPT consistency — Manakul et al.; the cheap single-pass hidden-state approximation — semantic entropy probes, Kossen et al.), building on Chapter 21’s trial series, not a single-run logprob dip. Those signals detect a probable confabulation at response level; they do not localize the wrong span or explain the mechanism, and they miss confident fabrication where every sample agrees on the wrong answer — which is the precise content of “signals never convict.” The attention and confidence caveats from Chapter 3 still hold.
• Forward link: probes fix incidents; they do not survive upgrades. The next chapter converts located, probed behaviors into assets that outlive the revision that produced them.

Next

The veer is located, probed, repaired — for this revision. Next quarter the model upgrades, the endpoint migrates, the template bumps, and the same refund question fails in a new place with new signals. Incident repair without upgrade discipline is rent, not ownership. The next chapter builds the asset that survives revisions: the behavioral diff.