Part II — Debugging Deterministic Software

Green here, red there, same code, same data

Every contract from Chapter 8 is in place. The suite passes on the engineer’s laptop. It fails in CI — same commit hash, same fixture file, same command. The failure is the old ghost from Chapter 4:

# laptop (passes)                    # CI (fails)
$ python jobs/run_invoices.py        $ python jobs/run_invoices.py
report.csv written, 312 rows         Traceback (most recent call last):
                                       File "billing/totals.py", line 88, in summarize
                                         row["refund_id"] = order["refund_id"]
                                       KeyError: 'refund_id'

git rev-parse matches. sha256sum fixtures/incident_input.csv matches. The engineer re-reads totals.py, re-inspects state, re-runs the boundary table — all clean, because the code is clean and the data is clean. Chapters 5–8 have nothing left to convict. That is precisely the signal: when identical code plus identical data still diverges, the divergence lives underneath both.

OBSERVATION: same code hash + same input hash → passes on laptop, raises KeyError in CI. HYPOTHESIS H1 (code/data defect): a latent code or data break that laptop runs mask (stale cache, dirty tree, uncommitted file). HYPOTHESIS H2 (environment divergence): interpreter, dependency, OS, config, or secret differs between the two runtimes. INFERENCE: none yet — the symptom is identical to Chapter 5’s, but Chapters 5–8 probes all pass here; only a substrate swap separates H1 from H2.

This chapter’s question: when code and data are pinned identical and outcomes still differ, what ordered probe convicts the environment?

Why environment bugs outlast their welcome

Three habits make them the longest-lived defects in deterministic software:

1. “Nothing changed.” The engineer means no code changed. But pip install resolved pandas 2.x in CI while the laptop runs pinned 1.5; the CSV reader’s dtype inference changed between those versions, and the refunds join column parses as NaN instead of strings on one side. Chapter 4 named this H3. Nothing in the diff changed; everything around the diff did.
2. Dirty-tree alibis. The laptop passes because of an uncommitted local edit, a .env file absent in CI, a PYTHONPATH entry pointing at last month’s billing/ package, or a warm __pycache__ / wheel cache. H1 (masking) is the reason the environment probe must start from a clean, locked rebuild — otherwise every “environment” conviction is really an uncommitted-code confession.
3. Config blindness. Secrets, env vars (REFUND_SOURCE=legacy vs. v2), OS line endings, timezone defaults, locale collation, file-path case sensitivity (Billing/ vs. billing/ on macOS vs. Linux), GPU-vs-CPU numeric paths. None appears in git diff. All appear in a parity checklist — if you write one before you need it.

The mental model is Chapter 4’s stack bottom, now with teeth: the environment is every input your program consumes that is not the code text and not the named data file. If it can differ between two runs, it is environment until proven otherwise. The probe discipline from Chapter 4 applies at full strength: hold code and data constant, swap the substrate, observe.

The method: the locked-container probe

Order matters — cheapest exoneration first, and H1 (dirty tree) before H2 (true drift), because a dirty tree makes every later probe lie:

    flowchart TD
    A{"tree clean both sides? (status / stash / rev-parse)"}
    A -->|no| H1["H1: dirty-tree / masking — clean and re-run before any env claim"]
    A -->|yes| B["freeze inputs: sha256 every fixture and config"]
    B --> C["pin randomness and time: seed, PYTHONHASHSEED=0, TZ=UTC"]
    C --> D["locked rebuild from the lockfile, caches disabled"]
    D --> E{"sides still differ?"}
    E -->|no| H1b["H1: the pass was a cache / stale-env artifact"]
    E -->|yes| F["bisect the freeze and env diff — one entry per run until the outcome flips"]
    F --> H2["H2: the flipped entry is the drift — pin it (lockfile / image / loader guard)"]
  

1. Prove the tree clean. git status --porcelain empty, git stash list empty, git rev-parse HEAD identical on both sides. Record pip freeze (or uv pip freeze / lockfile hash) on both sides. If the laptop tree is dirty, H1 is still live — clean it and re-run before any environment claim.
2. Freeze the inputs. sha256sum on every fixture and config file consumed by the repro, both sides. Chapter 3’s freezing rule extends below the code: hash the .env, the config/*.yaml, the fixture directory.
3. Swap the substrate, one variable per run. The discriminating experiment is the locked rebuild: run the pinned repro (same code hash, same input hash) inside a fresh container/image built from the lockfile — no caches, no local site-packages. Prediction if H1 (masking/latent code-data): the locked container fails everywhere (laptop’s pass was the artifact). Prediction if H2 (environment): the locked container passes or fails identically on both sides — the side-to-side difference disappears once the substrate is identical, and the pip freeze diff names the suspect package.
4. Bisect the diff. If H2 is supported, the pip freeze / lockfile diff plus env diff is the suspect list. Change one entry per run (downgrade one package, set one var) until the outcome flips. The flipped entry is convicted; everything else is context.

# parity probe transcript (commands, not results — reader executes)
$ git rev-parse HEAD && git status --porcelain        # must be clean + identical
$ sha256sum fixtures/incident_input.csv config/*.yaml
$ pip freeze > evidence/freeze-laptop.txt             # vs. evidence/freeze-ci.txt
$ diff evidence/freeze-laptop.txt evidence/freeze-ci.txt
$ docker build --no-cache -t repro:locked . && docker run --rm repro:locked python jobs/run_invoices.py
# Prediction H1: locked run fails (laptop pass was dirty/cache).
# Prediction H2: locked runs agree with each other; freeze diff names pandas 1.5 vs 2.x.

OBSERVATION (constructed illustration, not a measured run): laptop tree clean, input hashes match, freeze diff shows pandas==1.5.3 vs. pandas==2.1.0; both locked-container runs agree (fail identically once the CSV parse path is pinned to 2.x semantics). UPDATED BELIEF: H2 supported — dependency drift in CSV dtype inference; the laptop’s pass was a stale 1.5 environment, not evidence of code correctness. H1 suspended (tree was clean), but the cache audit stays in the checklist for next time. INFERENCE: the fix is a lockfile pin plus a parse-explicit dtype= / keep_default_na=False contract at the loader (a Chapter 8 guard at the data-entry handoff) — not a code-logic change.

Randomness and time deserve one line each: unseeded random/numpy/PYTHONHASHSEED differences mimic environment divergence perfectly — pin seed + PYTHONHASHSEED=0 before claiming drift. Clock/timezone differences (TZ, DST boundaries, datetime.now() in fixtures) do the same — pin TZ=UTC and freeze time in the repro. Both go on the checklist below as first exclusion rows.

Luo and colleagues catalogued the causes of non-deterministic test failures across 201 flakiness-fixing commits and found eleven recurring categories, dominated by async waits, concurrency, and test-order dependency, with unseeded randomness, time, network, floating-point, and platform dependence close behind (Luo et al., 2014). That taxonomy sharpens this chapter’s H1/H2 split. Some categories — platform dependence, filesystem semantics — are genuine substrate (H2). Others — async, concurrency, order dependency, unseeded RNG — are latent non-determinism in the code that a different substrate merely exposes (H1 in disguise). If the locked-container rebuild still disagrees run-to-run on the same machine, you are looking at one of the H1-flavored categories, not drift.

Research lineage: config and drift are where the outages live

Misconfiguration is a first-class failure class, not a footnote. Yin and colleagues studied 546 real misconfigurations — 309 from a commercial storage product deployed at thousands of sites, 237 from CentOS, MySQL, Apache, and OpenLDAP — and found that 70–85% were parameter mistakes, of which roughly 40–50% set a value that plainly violated a documented format or rule, and 12–30% were inconsistencies between two parameters that individually looked fine (Yin et al., 2011). Two lessons for this chapter. First, “config blindness” is not a minor habit — configuration is a leading root cause of production incidents. Second, a large fraction of config errors are mechanically checkable, which means the fix is a Chapter 8 contract at the config-entry boundary: validate REFUND_SOURCE in {"legacy", "v2"} at startup, and the next bad value fails loudly instead of silently selecting the wrong code path.

Containers are the parity mechanism, with caveats. Building the repro from a lockfile in a --no-cache image is the practical form of reproducible-environment research (Boettiger’s account of Docker for reproducible work is the canonical statement), but a container pins the userland, not the kernel, the CPU, or the GPU (Boettiger, 2015). Numeric divergence from CPU-vs-GPU dispatch or floating-point associativity survives an identical image. Frameworks expose determinism switches for this — torch.use_deterministic_algorithms(True), the cuDNN benchmark flag, CUBLAS_WORKSPACE_CONFIG — but they cost throughput, they do not make results match across different GPU models or driver versions, and some operations have no deterministic implementation at all. So “identical locked container” is a weaker guarantee for numerical code than for the deterministic pipelines of Part II.

And the ground shifts under all of this in Part III. The environment-parity discipline assumes that once the substrate is identical the outcome is identical. Machine-learning systems break that assumption even inside one locked container — Chapter 12 makes reproducibility a designed property rather than an assumed one, Chapter 14 handles device/precision divergence at the tensor boundary, and Chapter 21 treats sampling nondeterminism (including server-batch effects that persist at temperature 0) as a debugging object in its own right. The reproducibility literature in AI — few papers ship runnable code, fewer reproduce independently — is the organizational symptom. Part II’s promise — pin everything, get the same answer — is the last time this book can make it cleanly.

Lab 9: isolate env vs. code with one locked run

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own parity record.

Setup. Take any passes-here/fails-there repro (use the pandas-drift pair above, or inject one: install two dependency versions, set a divergent env var like REFUND_SOURCE, or run with vs. without a local .env). You need two runtimes (laptop vs. CI, two venvs, or venv vs. container) with identical code and data hashes.

Task.

1. Write H1 (dirty-tree/masking: one side’s outcome is the artifact) and H2 (substrate drift: runtimes genuinely differ) with distinct locked-run predictions before rebuilding.
2. Complete the parity checklist (paper form below) on both sides: code hash, tree cleanliness, input hashes, freeze files, env diff, seed/TZ pins.
3. Run the locked rebuild (--no-cache, from lockfile). Independent variable: runtime substrate (side A vs. side B vs. locked). Controlled variables: code hash, input hashes, command line, seeds.
4. Record OBSERVATION per runtime (pass/crash verbatim + traceback line if any) and UPDATED BELIEF. If H2 wins, bisect the freeze/env diff one entry per run until the flip.

Success criterion. A completed parity record plus the convicted entry (package == version, env var, or dirty-tree confession) and the lockfile/config change that makes both sides agree. A rebuilt image without the before/after freeze diff is not completing the lab — the diff is the diagnosis.

Companion tool: Environment Parity Checklist

What it accepts: the pinned repro reference (code hash + input hashes), the two pip freeze / lockfile artifacts, the env diff, seed/TZ pins, and the three-row runtime table. What it performs: it enforces exclusion order — dirty-tree and cache checks before any drift claim, seed/TZ pins before package bisection — and it refuses an H2 verdict while any checklist row is still UNKNOWN. It records the convicted entry with its flip demonstration (the one-entry change that moved the outcome). What it can establish: which substrate entry explains the side-to-side divergence under the pinned repro — or that no substrate entry does (both locked runs still disagree → the “identical” inputs were not identical; re-audit hashes and caches). What it cannot establish: code/data correctness (a converged environment can still carry a Chapter 5–8 defect — parity is necessary, not sufficient), intent truth (no spec line → UNKNOWN), or future stability (today’s lockfile rots tomorrow without CI enforcement). How its output changes your next action: a convicted entry routes to pinning it (lockfile, Dockerfile, dtype= guard, .env.example) with the parity record as the regression artifact; a dirty-tree confession routes to clean-tree discipline (pre-commit hooks, git status in CI); persistent disagreement routes back up the stack to Chapters 5–8 with the environment exonerated in writing.

Paper form, sufficient for this chapter:

Repro: code ___ | inputs ___ | cmd ___
Tree clean (both sides)? Y/N   Caches disabled (--no-cache / fresh venv)? Y/N
Seeds pinned (seed=___, PYTHONHASHSEED=___)? Y/N   TZ=___?
Freeze diff suspect(s): ___
Locked run A: ___  Locked run B: ___  (agree? Y/N)
CONVICTED ENTRY: ___  FLIP DEMO (one-entry change → outcome): ___
PIN APPLIED: lockfile / image / dtype-guard / env-example (circle)

Where a software implementation does not yet exist in the reader’s stack, this checklist is the tool. The parity discipline precedes any automation.

Reusable procedure: every “works on my machine” gets this

1. Pin code + data (hashes both sides) before touching anything.
2. Confess or exonerate the tree (status, stash, caches, .env, PYTHONPATH).
3. Pin randomness and time (seed, PYTHONHASHSEED, TZ=UTC).
4. Diff the substrate (freeze files, env, OS/image tags).
5. Locked rebuild, --no-cache, from the lockfile; compare three outcomes.
6. Bisect one entry per run to the flip; pin the conviction (lockfile + loader guard + CI check).

Failure modes

• Diff-first debugging. Reading code for an hour before checking git status and pip freeze. The cheapest probes are below the code — run them first.
• Cache-forged passes. A warm Docker layer, a stale venv, or __pycache__ masking the break on one side. --no-cache / fresh-venv is not hygiene theater; it is the experiment’s control.
• Unpinned floating deps. pandas>=1.5 resolving differently per month per machine. Floating specifiers convert every install into a potential H2.
• Env-var oral tradition. REFUND_SOURCE set in one engineer’s shell profile, absent everywhere else. Undocumented vars are unversioned code — promote them to .env.example + CI assertions.
• Agreement-as-parity. “Both sides run Python 3.11, so environments match.” Minor versions, patch releases, build flags, and transitive deps all diverge beneath the headline version. The freeze file is the environment; the headline is a rumor.
• Single-run exoneration. One locked pass declaring the environment clean forever. Lockfiles rot; today’s parity proves today’s run. CI rebuilds from the lockfile on every change — that is the Chapter 8 tripwire for this layer.

Limits, per contract: one parity record convicts one entry under one repro; it does not certify code/data correctness, does not freeze the future, and does not replace human verification where money, safety, or production traffic is at stake. UNKNOWN where any checklist row is unmeasured.

References

• Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. An Empirical Analysis of Flaky Tests. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014, pp. 643–653. https://doi.org/10.1145/2635868.2635920
• Zuoning Yin, Xiao Ma, Jing Zheng, Yuanyuan Zhou, Lakshmi N. Bairavasundaram, and Shankar Pasupathy. An Empirical Study on Configuration Errors in Commercial and Open Source Systems. Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP), 2011, pp. 159–172. https://doi.org/10.1145/2043556.2043572
• Carl Boettiger. An Introduction to Docker for Reproducible Research. ACM SIGOPS Operating Systems Review 49(1), 2015, pp. 71–79. https://doi.org/10.1145/2723872.2723882

Debugging Checklist

• Code + input hashes verified identical on both sides before any theory?
• Tree clean, caches disabled, .env/PYTHONPATH audited (H1 exonerated or confessed)?
• Seed, PYTHONHASHSEED, TZ pinned?
• Freeze + env diffs recorded with named suspects?
• Config values validated at the entry boundary (Ch 8 contract), not just read?
• Locked --no-cache run compared across runtimes; H1/H2 predictions pre-written?
• Same-machine repeat of the locked run checked (rules out async/order/RNG non-determinism)?
• Conviction demonstrated by a one-entry flip; pin applied (lockfile/image/guard/env-example)?
• No headline-version agreement or single-run outcome treated as parity?

What This Chapter Established

• Environment as the stack bottom: every non-code, non-data input is substrate until proven otherwise, with dirty-tree/masking (H1) excluded before drift (H2) is claimed.
• The locked-container probe with exclusion order (tree → hashes → seeds/time → freeze/env diff → --no-cache rebuild → one-entry bisection), demonstrated on the pandas 1.5-vs-2.x CSV-inference drift — constructed illustration, no measured runs claimed.
• Empirical grounding: non-determinism has a taxonomy (Luo et al. — async, concurrency, order, RNG, time, platform), and its categories split along the H1/H2 line; misconfiguration is a leading outage cause and largely mechanically checkable (Yin et al.), so config gets a Ch 8 boundary contract; containers pin userland but not kernel/CPU/GPU (Boettiger).
• Lab 9 as a proposed parity record the reader executes; the Environment Parity Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: code/data correctness under a converged environment, future stability without CI enforcement, or any parity claim resting on headline versions or warm caches.
• Part II’s closing map: Chapter 5 reads the traceback, Chapter 6 inspects live state, Chapter 7 hunts boundaries, Chapter 8 pins handoffs with contracts, this chapter pins the substrate — deterministic debugging, end to end.

Next

Part II assumed a program you can re-run: same code, same data, same container, same outcome — or a substrate diff that explains the difference. Part III breaks that assumption on purpose. The notebook on your screen is not the program that ran: cells executed out of order, hidden state carried between runs, and “Restart & Run All” tells a different story than the session you debugged. The next chapter opens that gap — the notebook is not the program you see.