← Adversarial Math From First Principles

Provenance and Reproducibility

Every result carries replayable provenance: what is recorded, how fresh runs are compared to committed ones, and the reproduction gap that remains.

A theorem-prover result that exists only on one laptop is hard to trust. Here are four ordinary ways it fails to travel:

  1. The proof depended on a Mathlib version nobody wrote down.
  2. The source file hashes differently on Windows and Linux because of line endings.
  3. A report was edited by hand “just to fix a typo” after the run.
  4. A stale result file from an earlier run is still in the output directory and gets read as current.

The repository has a defence against each, and this chapter walks through them: what every result records, how a fresh run is compared with the committed one, and what has still not been reproduced. It builds on claims (Chapter 1) and the evidence statuses (Chapter 3). The project also keeps its own failed runs on record, because a reproducibility story with no failures in it is probably missing some.

What each result records

An EvidenceResult carries the claim ID, version and fingerprint; the system and tool version; the full command; start and finish times; the encoded domain; the assumption IDs; inputs and outputs; and the unresolved obligations and limitations. Kernel results add the resolved executable path, stdout and stderr, the exit code, and the SHA-256 of the checked source.

Two choices keep those hashes stable.

  • Line endings are normalized. Source hashes are computed over UTF-8 text with CRLF turned into LF, and .gitattributes pins LF checkout for formal sources. Without that, the same proof hashes differently on Windows and Linux. The rule later had to be extended to P7’s JSON fixtures: on a fresh Windows checkout they came out with CRLF and failed the fixture check until experiments/repairs/** text eol=lf was added.
  • The environment is pinned. uv.lock is committed and CI checks that it is unchanged; lake-manifest.json pins Mathlib, and CI never runs lake update.

Why a hash and not a date

A timestamp says when a file was made. A hash says which bytes. When the book cites formal/lean/EuclidI47Coordinate.lean as b5b3a709…f688 (its LF-normalized SHA-256), a reader can check the bytes they hold against it, whatever the date.

The citation rule for notebooks goes further. It lists the plan ID, the index ID, the dossier IDs, the source and executed notebook hashes, the canonical executed hash, the execution record ID and the resolved package versions, because no single identifier pins the whole chain.

Three identities, not one

When are two runs “the same result”? The project’s contracts (from its P1 phase) distinguish three answers:

Identity Covers Changes when
Semantic identity Everything that bears on meaning; excludes timestamps, durations, stdout/stderr, working directories, absolute paths, solver statistics A claim, source hash, status or conclusion changes
Full content hash The whole record except its own hash Anything changes, including a duration
Language canonical message hash The Language Runtime’s normalized form Not interchangeable with the others

This is why two runs on different days can be “the same result” and still differ byte for byte. In the P4 record, two clean builds produced identical source bytes, identical executed bytes and identical semantic IDs, but different full index hashes, because the index covers its generation time. That is by design, not a defect.

Fresh run versus committed run

scripts/ci_report_consistency.py regenerates the report from scratch in CI and compares it with the committed artifacts/math01/report.json, after removing volatile fields. Its tests tamper with a report in the ways that matter and expect failure:

  • a changed source hash;
  • an inflated status;
  • a missing required result;
  • a simulated execution;
  • a changed claim fingerprint;
  • a removed obligation;
  • a changed counterexample assignment.

A separate test checks the other direction: volatile fields alone (timestamps, paths, durations) must not fail the comparison.

Stale files are handled by the --clean flag, which removes old report, claim and result JSON before writing, so a regeneration cannot leave orphaned evidence behind. Accepted evidence is never overwritten in place. The original R1 bundle is preserved byte for byte under artifacts/history/math01-r1/, separately from the current baseline, and each later evidence record restates the hash of the accepted report to show it is unchanged.

Keeping the failures

The R2 hosted qualification, the first full run on GitHub’s machines, took seven attempts. The first six failed for reasons such as running the axiom audit before the build, losing a log because the upload was skipped on failure, and repeated trouble with the Mathlib cache on Windows. The fix that finally worked moved the Lean lanes to Ubuntu. The project’s record says these runs exposed “CI integration defects” and did not alter the claim contracts or weaken any control.

Later failures are kept the same way: an R4 run that failed because a Rocq file was not yet built before the statement audit, and a P8 run that ran out of disk space. In each case the failing run stays in the record instead of being replaced by the passing one. The details are in the evidence record below.

What has not been reproduced

The project’s status document is blunt:

  • “Clean-machine reproduction outside GitHub-hosted runners was not performed.”
  • The R1 run “was executed in a prepared Windows worktree, not from an empty machine.”
  • The P8 external build ran locally in WSL with landrun --best-effort, “not a container or a full VM,” and its first hosted attempt failed on disk space.
  • Opam warning cleanup, dependency caching and optional containerization are deferred.

“Hosted-qualified” means a GitHub-hosted runner reproduced the result. That is a stronger claim than “it worked on my machine” and a weaker one than “an independent party reproduced it”. The independent-reproduction review dimension is not_evaluated throughout. Chapter 29 returns to this.

What this establishes

Reproducibility here means that a stated procedure, applied to stated inputs with stated tools, gives a record with the same semantic identity. It does not mean the result is true. A perfectly reproducible wrong claim is still wrong.

Reproducing a result on a second machine is also different from reproducing it on a second kernel. This chapter covers the first; Chapter 15 covers the second.

Try it yourself

uv sync --all-groups --locked
uv run math-engine math01 run --clean --output artifacts/math01
uv run pytest
uv run ruff check .
uv run python scripts/ci_negative_controls.py
uv run python scripts/ci_report_consistency.py --validate-only artifacts/math01/report.json

For the Lean and Rocq lanes, follow docs/methodology/reproducibility.md: Lean 4.35.0 with Mathlib v4.35.0, and Rocq 9.0.1. It warns that lake update changes pins and should be run locally only when reviewing a dependency change.

Seeded error. Hand-edit one status in a copy of artifacts/math01/report.json (for instance, change a not_evaluated to formally_verified) and run ci_report_consistency.py --validate-only on it. Expect failure: status inflation is one of the mutations the tests cover.

Open questions.

  1. What is the minimum an outside party needs to reproduce MATH01 from nothing? Today that includes a pinned Lean toolchain, the Mathlib cache and Rocq 9.0.1. Is a container image part of the claim?
  2. The semantic-identity exclusion list is a policy. Which excluded field, if it changed, could hide a real difference?
  3. Hosted runners can change under the project. Which results would silently change if GitHub changed its Ubuntu image, and would a check catch it?

Evidence record

The accepted report artifacts/math01/report.json has SHA-256 ba0eee5af97fde2e5f569c76b5bfac0df66b8fc2a88840f3979b2e2d853995f9; the P4 to P8 records each restate it.

The R2 hosted qualification, run by run:

Run What failed Fix
1 Lean axiom audit ran before lake build Build first
2 Formal-audit log lost: upload skipped on failure Failure-preserving uploads
3 Windows Mathlib cache extraction incomplete Retry logic
4 Report job’s Windows cache build failed Reordered unpack and build
5 Same cache problem on C: Forced unpack
6 Forced unpack still failed Moved Lean lanes to Ubuntu/elan, Rocq to opam
7 Pass, then post-merge main also passed

Later preserved failures: R4 run 40152280148 (and the post-merge run on the same head) failed because PhysicalComposition.vo was not yet built before the statement audit; the first P8 hosted attempt 40562688741 ran out of disk space during the external build.

Hosted qualification: R2 PR run 40036592465 and post-merge main run 40045572945. Later phases carry their own run IDs in their evidence records, for example P3 post-merge 40227260682, P5 40320278433, P6 40400910044 and P7 40470483277. This chapter quotes those IDs and has not looked up the runs.

Sources

Toolchain documentation (citation class software_documentation):

The book does not yet cite the reproducibility literature; that remains open. In the repository: docs/methodology/reproducibility.md, docs/research/MATH01-status.md, docs/research/MATH01-R2-hosted-run-7.md, scripts/ci_report_consistency.py and .github/workflows/math01.yml.