Part IV — Debugging Models

The case of the vanishing exception

Chapter 19 certified the rendered bytes — section 4.2 present, ids round-tripped. Yet the refund failure persists on long cases: short split-shipment queries pass, the same question with a full ticket history fails. The bytes were complete at render time and incomplete at generation time. Between the two stands a silent editor: the context window.

Concrete failure. A 14,000-token assembled context (system + six retrieved sections + two years of ticket history + the question) enters a pipeline whose effective limit is 8,000 tokens. The assembler truncates from the middle — history kept, question kept, the retrieved exception dropped without a log line. The short fixture never notices; production always does.

OBSERVATION: pass rate falls with context length (MEASUREMENT on the reader’s own fixture: short ≤2k tokens vs. long ≥10k tokens, same question class, same revision, temperature 0, ≥5 trials each). HYPOTHESIS H1 (tail cut): the decisive section sits past the limit and is cut — truncation from the end (or middle) removes 4.2. HYPOTHESIS H2 (order burial): 4.2 survives but lands where the model under-uses it — middle-positioned decisive content ignored despite presence. HYPOTHESIS H3 (budget starvation): the input fits but leaves no output budget — generation stops mid-citation (max-tokens cut), and the “wrong answer” is an unfinished one. INFERENCE: none yet — only length accounting per segment plus truncation-side probes separate a cut from a burial from a starved finish.

This chapter’s question: what got cut, dropped, or reordered before generation — and which side of the window owns the failure?

Why “raise the limit” fails first

The obvious move — switching to a larger context window (or a model advertising one) — fails as diagnosis because limits are per-deployment facts, not brochure facts, and truncation policy is code, not physics. Three length traps:

1. Advertised vs. effective limit. The headline window, the endpoint’s configured max, the assembler’s own cutoff, and the reserved output budget are four different numbers. Debugging against the brochure measures nothing. And the behavioral effective limit is lower still: Hsieh and colleagues’ RULER benchmark evaluated 17 models claiming 32k+ context windows and found only about four sustained acceptable performance at 32k, with large accuracy drops well before the nominal limit — despite near-perfect scores on the simpler needle-in-a-haystack test (Hsieh et al., 2024).
2. Silent truncation. Most assemblers cut without marking: no <truncated> flag, no dropped-segment log, no error. Absence of evidence becomes evidence of completeness — the precise inversion Chapter 19 was built to prevent.
3. Position without presence confusion. Even uncut context is not uniformly used: decisive content buried mid-window underperforms the same content at the edges in many observed setups (Liu and colleagues’ “lost in the middle,” Chapter 18) — but the magnitude is an empirical property of the reader’s own fixture and revision, not a universal law citable from this book. The needle-in-a-haystack test that reads near-perfect is known to overstate real long-context use: single-needle retrieval is the easy case, and performance falls once the task needs multiple facts, reasoning across them, or has semantically similar distractors nearby. Measure per case, never assume.
4. History-growth blindness. The context that fit in testing grows in production as conversation history, tool traces, and retrieved sets accumulate. Length is tested at t=0 and violated at t=40 — the ledger must cover the longest realistic assembly, not the demo’s.

OPINION: every context pipeline needs length accounting the way every budget needs a ledger — per segment, in tokens, with the truncation rule written down. Length is the cheapest measurement in Part IV and the most skipped. Teams that ledger length stop filing “model got dumber” tickets for arithmetic.

The mental model: the window as a ledger with a truncation policy. Tokens in (per segment) vs. tokens allowed (effective input budget) vs. tokens reserved (output budget) vs. policy (which end dies first, what marker is left). The first divergence is arithmetic: the ledger either balances or it names the dropped row.

The method: truncation forensics

1. Ledger the lengths. Token-count every segment with the deployment’s tokenizer; record the effective input limit, the reserved output budget, and the assembler’s truncation rule (head/middle/tail, per-segment priorities). Hash the ledger into the bundle. Any of these numbers taken from marketing copy instead of measured configuration is UNKNOWN.
2. Mark the cut. Re-render with truncation markers enabled (or simulate the policy offline): which segments shrink, which vanish, in what order. The missing 4.2 appears here as DROPPED, with the policy line that dropped it.
3. Run truncation-side probes. (a) Shortening probe: strip low-priority history to fit 4.2 inside budget — prediction if H1: failure flips to pass; (b) Reorder probe: same tokens, 4.2 moved to the front (or end) — prediction if H2: pass without removing anything; (c) Budget probe: same input, output budget doubled — prediction if H3: truncated citations complete. One probe per run, ≥5 trials, FORECASTs pre-written.
4. Pin the policy. The surviving probe earns a ledger assertion (per-segment budgets + truncation rule + marker requirement) enforced in CI.

    flowchart TD
    L["ledger every segment in tokens with the deployment tokenizer"] --> BAL{"total <= effective input budget?"}
    BAL -->|no| SIM["simulate the truncation policy: name the DROPPED rows"]
    BAL -->|yes| POS["everything fits — position or output budget still suspect"]
    SIM --> SH["shorten probe: strip low-priority history so 4.2 fits (x5)"]
    POS --> RO["reorder probe: same token multiset, 4.2 moved to the front (x5)"]
    POS --> BU["budget probe: same input, output budget doubled (x5)"]
    SH --> V{"which probe flips the failure?"}
    RO --> V
    BU --> V
    V -->|"shorten only"| H1["H1: tail / middle cut — pin per-segment budgets, make 4.2 non-truncatable"]
    V -->|"reorder only"| H2["H2: mid-window burial — ordering policy + position tests"]
    V -->|"budget only"| H3["H3: output starvation — reserve output budget, add finish detection"]
  

# truncation forensics (arithmetic before theories)
ledger = account_tokens(segments, tokenizer=deploy_tokenizer)  # per-segment tokens
print(ledger, "effective_in:", EFF_IN, "reserved_out:", RES_OUT, "policy:", TRUNC_POLICY)
marked = simulate_truncation(ledger, EFF_IN, TRUNC_POLICY)      # which rows die first?
print("dropped:", marked.dropped, "shrunk:", marked.shrunk)
# Probes, one per run, >=5 trials, temperature 0:
print("shorten:", run(bundle.strip("history", to_fit=True), trials=5))   # H1
print("reorder:", run(bundle.move("4.2-exception", to="front"), trials=5))  # H2
print("budget:", run(bundle.with_params({"max_tokens": 2000}), trials=5))   # H3
# FORECASTs: H1 flips only shorten; H2 flips only reorder; H3 flips only budget.

OBSERVATION (constructed illustration, not a measured run): ledger 14,203 tokens vs. effective input 8,000; policy middle-truncate drops 4.2 (rank 3 of 6 docs); shorten probe 10/12, reorder probe 4/12 on the long bundle, budget probe 3/12 with completions still missing the citation. UPDATED BELIEF: H1 supported for long cases; H2/H3 suspended here — order and budget move nothing while the section is physically absent. INFERENCE: fix per-segment budgets + priority pinning (4.2 never truncatable) + truncation markers; short-fixture passes were measuring a different input than production sends.

Note the confound the three probes control for: shorten and reorder both change something about position, so only their disagreement separates a cut from a burial. Shorten flips while reorder holds flat means the section was absent, not ignored — presence first, position second. Had both flipped, the verdict would be UNKNOWN with a combined probe (shorten + reorder independently) named, not a victory lap.

Research lineage: the window has three different edges

The behavioral edge is measured, and it is short. RULER’s headline result — most models claiming 32k+ degrade sharply before 32k — means the shorten probe (H1) and a length sweep are complementary. Even with 4.2 physically present and uncut, a model may fail to use it at 12k tokens while succeeding at 4k. The chapter’s own advice to test across the deployed length distribution is RULER’s methodology at fixture scale (Hsieh et al., 2024). And when the reorder probe (H2) convicts a mid-window burial, there is a mitigation that lives outside the weights: the inference-time attention calibration from “Found in the Middle” (Chapter 18) subtracts the model’s positional bias, recovering much of the lost accuracy without moving a token.

The needle test flatters the pipeline — and so, largely, does RULER. A pipeline that passes an internal needle-in-a-haystack check can still fail production, because production asks the model to combine several retrieved facts and reason over them — the regime where long-context performance drops. Yen and colleagues’ HELMET benchmark makes the point at scale: synthetic recall tasks (NIAH, and to a large extent RULER-style probes) saturate quickly or sit near zero, giving poor separation between models, while realistic composite tasks — retrieval-augmented QA, multi-document reasoning, many-shot in-context learning, summarization — are where the persistent degradation and the lost-in-the-middle effect actually show up (Yen et al., 2025). Do not accept a green synthetic score — even a RULER number — as evidence that length is not the problem; the length sweep should use question shapes that match production.

Sometimes the window is architecture, not policy. Models using sliding-window attention or streaming schemes (attention sinks, as in Xiao and colleagues’ StreamingLLM) structurally cannot attend to tokens far enough back, regardless of the assembler’s truncation rule (Xiao et al., 2024). When the ledger balances and the shorten probe still flips, the “effective input limit” you should record may be an architectural attention span, not a configured cutoff.

Lab 20: length ledger with truncation-side probes

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own ledger and probe table.

Setup. Take one length-sensitive failure (or inject one: pad a passing case with history until it exceeds the effective limit). Measure the effective input limit and output reservation from the reader’s own deployment configuration — not from any vendor headline.

Task.

1. Write H1/H2/H3 with distinct numeric FORECASTs before probing (e.g., “H1: shorten ≥10/12, reorder ≤5/12; H2: reorder ≥10/12 with identical token multiset; H3: budget-double completes citations ≥10/12”).
2. Independent variable per run: exactly one length intervention (content removed, order changed with tokens held, or output budget changed). Controlled variables: revision, seed, question text, temperature 0.
3. Build the ledger (per-segment tokens, limits, policy), run all three probes (≥5 trials each), record OBSERVATION (scores + dropped-row lists) and UPDATED BELIEF per row. Mixed patterns are UNKNOWN with the confounded leg named.
4. Write the ledger assertion on paper: per-segment caps, priority order, truncation markers, CI length gate.

Success criterion. A completed ledger + three probe rows matching one pre-written pattern, plus the written ledger assertion. A passing long case without the ledger is explicitly not completion — length luck is not length discipline.

Companion tool: Context Window Inspector

What it accepts: the per-segment token ledger, the effective input/output budgets with their configuration sources, the truncation policy + marker setting, and the three probe series with FORECASTs. What it performs: it verifies ledger arithmetic (sums vs. budgets), simulates the cut order, checks probe outcomes against FORECASTs, requires ≥5 trials per stochastic probe, refuses a truncation verdict while the policy or budgets are unsourced, and stamps the ledger assertion with hashes. What it can establish: what was cut, shrunk, or starved — and which side (input cut, position burial, output starvation) owns this fixture’s failure under this deployment. What it cannot establish: cross-deployment generality (budgets and policies differ per stack), position-effect universals (measured here, not cited as law), or provider-side truncation it cannot observe — those stay UNKNOWN. It never treats the advertised window, a single long-case pass, or a model self-report as length evidence. How its output changes your next action: an H1 conviction routes to segment budgets + priority pinning + markers; H2 routes to ordering policy + position regression tests; H3 routes to output-budget reservation + finish-detection; a clean ledger with sustained failure routes to Chapter 21 — length certified, sampling next.

Paper form, sufficient for this chapter:

LEDGER: sys __tok | docs __tok | hist __tok | q __tok | TOTAL __tok vs EFF_IN __tok (source: ___)
POLICY: ___ (head/mid/tail) | markers ON/OFF | RES_OUT ___ (source: ___)
SHORTEN ×5: ___   REORDER ×5: ___   BUDGET ×5: ___   CONVICTION: H1 / H2 / H3 / UNKNOWN
ASSERTION: caps ___ | priorities ___ | marker req ___ | CI gate ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The ledger discipline precedes any automation.

Reusable procedure: every long-case failure gets this ledger

1. Source the budgets — effective input, reserved output, from deployment config, never headlines.
2. Ledger the segments — tokens per segment with the deployment tokenizer.
3. Simulate the cut — policy applied on paper; dropped rows named.
4. Probe three sides — shorten, reorder, budget; one per run, ≥5 trials.
5. Assert the ledger — caps, priorities, markers, CI gate.

Failure modes

• Brochure budgeting. Sizing contexts against the advertised window. The effective limit is a configuration MEASUREMENT or it is UNKNOWN.
• Silent-cut blindness. No markers, no dropped-row log. Every truncation without a marker is a future misattribution filed in advance.
• Short-fixture certification. Passing 2k-token tests and shipping into 14k-token production. Fixtures must span the deployed length distribution.
• Position folklore. “Models ignore the middle” (or “love the end”) cited as law. Position sensitivity is a per-fixture EXPERIMENTAL RESULT here, never a premise.
• Output-budget amnesia. Debugging the input while max-tokens strangles the answer. A citation cut mid-sentence is a budget failure wearing an ignorance costume.
• Markerless pipelines. Shipping an assembler with no truncation flag after this chapter. The next length incident then starts from zero instead of from the marker log — a solved problem re-rented.
• Single-length fixtures. Testing at one context size and extrapolating. Length behavior is a curve (short vs. mid vs. long); one point never draws it.

Limits, per contract: one ledger convicts one length cause under one deployment’s budgets, policy, and tokenizer; it does not generalize across stacks, does not explain uncut failures, and rots on any config or version change without re-ledgering. UNKNOWN wherever budgets are unsourced or probes ran single.

References

• Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the Real Context Size of Your Long-Context Language Models? Conference on Language Modeling (COLM), 2024. https://arxiv.org/abs/2404.06654
• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173. https://doi.org/10.1162/tacl_a_00638
• Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks. International Conference on Learning Representations (ICLR), 2024. https://arxiv.org/abs/2309.17453
• Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. International Conference on Learning Representations (ICLR), 2025. https://arxiv.org/abs/2410.02694

Debugging Checklist

• Effective input + reserved output sourced from deployment config (not headlines)?
• Per-segment token ledger built with the deployment tokenizer?
• Truncation policy written down; dropped/shrunk rows simulated on paper?
• H1/H2/H3 with distinct numeric FORECASTs before probing?
• Shorten / reorder / budget probes run, one per run, ≥5 trials?
• Length sweep run (short / mid / long) with 4.2 present and uncut throughout?
• Truncation markers and CI length gate specified?
• Budgets re-sourced from config after any deploy change (never carried over)?
• Cut simulation written on paper before the first probe (dropped rows named)?
• Short fixtures extended to cover deployed lengths?

What This Chapter Established

• Truncation forensics: length ledger per segment vs. sourced budgets, cut simulation, and three truncation-side probes (shorten/reorder/budget), demonstrated on the vanishing-4.2 case as H1 — constructed illustration, no measured runs claimed.
• Lab 20 as a proposed ledger-plus-probe record the reader executes; the Context Window Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any universal position law, any cross-deployment length claim, or anything about provider-side behavior beyond the reader’s own configuration.
• Research grounding: the behavioral effective context is measured and far below the advertised window (RULER — most 32k-claimed models degrade well before 32k); synthetic recall probes including NIAH and largely RULER overstate real long-context use, so the length sweep needs production-shaped tasks (HELMET); a mid-window burial (H2) has an inference-time fix outside the weights (“Found in the Middle”, Ch18); and the limiting edge can be architectural attention span, not a configured cutoff (StreamingLLM). A length sweep with the decisive content held present complements the shorten probe.
• Forward link: inputs can be complete, certified, and fully inside budget — and the answer still varies run to run. The next chapter treats that variance as the debugging object it is.

Next

The ledger balances: everything intended fits, nothing drops, markers confirm it. Yet the same frozen input yields Tuesday’s correct citation and Wednesday’s confident error. Nothing was cut — but something still varies, because sampling is not noise around the program. Sampling is part of the program. The next chapter debugs distributions.