Replay a hundred and three traces and check the answer again

The capstone replays the Chapter 12 condition ladder over 103 end-to-end traces. Replay mode is the important part: it reads the frozen trace rather than calling a model, so the result is deterministic and the run measures the memory, not the model that happened to be available on the day.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_capstone.py --mode replay

Python 3.11 or later. Replay mode makes no model call and needs no service. The run writes a new run directory under experiments/benchmark/runs/; a frozen run is never overwritten, and each run gets its own directory and manifest.

Expected outcome

capstone replay -> .../experiments/benchmark/runs/capstone-<timestamp>-replay
verdict: A
C4-C0: 0.2619  BR-BA: 0.5278

Three things in three lines. The verdict is the capstone’s own lettered judgement on the replay. C4-C0 is the gap between the fourth condition and the no-memory condition: what the memory added over having none. BR-BA is the gap between the best and the second-best condition, which is the number that says whether the ordering is stable rather than lucky.

The chapter’s own text reports the same two figures for the frozen replay, so the run here is the same measurement rather than a new one.

Why replay and not a fresh run

A fresh run would call a model, and the model would be a different model, at a different temperature, with a different prompt rendering. The result would still be a memory result, but it could not be compared with the earlier one without re-establishing everything the earlier run pinned. Replay removes the model from the comparison entirely: the traces are the input, the memory is the system under test, and the only thing that changes between two replays is the memory.

That is what makes the reproducibility address meaningful. A run id names a corpus version, a seed, a task set and the digests of everything the result depended on, so a later reader can check the inputs rather than trust the number.

Mechanism and limitations

Documented: the replay mode, the conditions, the verdict letters and the two reported gaps are in solution/run_capstone.py and the solution/capstone/ module. The chapter’s own text names the reproducibility address for the frozen replay.

Observed: the run above. The replay produced verdict A, a C4-C0 gap of 0.2619 and a BR-BA gap of 0.5278, and it wrote a new run directory with its own manifest.

Not shown. A replay measures the memory against traces that were recorded earlier, under conditions that were chosen then. It does not measure the memory against new traffic, and it does not measure whether the memory changed the behaviour that the traces recorded: behaviour.memory_changed_action and behaviour.memory_improved_action were pending in the measurement instrument, and a replay does not fill them in. The verdict letter is the capstone’s own rubric, and the gaps are comparisons between conditions that differ in more than one respect.

Canonical source and the capstone .