memory examples on Programmer.ie: Modern AI programming

Score three memory systems against the same six tasks

Mon, 01 Jan 0001 00:00:00 +0000

A memory book needs an instrument before it needs a memory. The chapter’s claim is that the measurement has to be separate from the thing being measured, and that a number without a failure attribution is not a result. This is that instrument: six canned tasks, three canned systems, and a scorecard that says which failures belong to which cause.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
cd experiments/benchmark
python -B -m memory_measurement.demo

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
cd experiments/benchmark
python -B -m memory_measurement.demo

Python 3.11 or later. The demo has no dependencies, calls no model, and needs no service. It scores canned responses against canned labels, so it is a test of the instrument, not of any memory.

Tell a current fact from a historical one

Mon, 01 Jan 0001 00:00:00 +0000

“Is it still true?” is two questions, and a memory that answers the wrong one is worse than one that answers neither. The suite here scores current state accuracy and historical state accuracy separately, per temporal shape, so a system that remembers everything and a system that only remembers the present are not averaged into one number.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch8.py --outdir ./runs/ch8-demo

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch8.py --outdir ./runs/ch8-demo

Python 3.11 or later. The suite is fixture-level and deterministic: zero model calls, no database, no service. It writes a new run directory; frozen runs are never overwritten.

Find the loops nobody wrote down

Mon, 01 Jan 0001 00:00:00 +0000

Some work is not finished and nobody wrote that down. The chapter’s claim is that a memory can infer those open loops, and that the interesting measurement is not how many it finds but how many it invents. The suite here reports both, and they are not the same number.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch9.py --outdir ./runs/ch9-demo

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch9.py --outdir ./runs/ch9-demo

Python 3.11 or later. The suite is fixture-level and deterministic: zero model calls, no database, no service. It writes a new run directory.

Staged gates beat one big prompt, and the ablation shows it

Mon, 01 Jan 0001 00:00:00 +0000

A derived loop is a consequence nobody wrote down: a change in one record that implies something about another. The chapter’s claim is that these are hard enough that the order of the gates matters, and the suite tests that by ablation: the same derived loops, with one stage removed at a time.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch11.py --outdir ./runs/ch11-demo

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch11.py --outdir ./runs/ch11-demo

Python 3.11 or later. The suite is deterministic with zero model calls. It writes a new run directory; frozen runs are never overwritten.

Replay a hundred and three traces and check the answer again

Mon, 01 Jan 0001 00:00:00 +0000

The capstone replays the Chapter 12 condition ladder over 103 end-to-end traces. Replay mode is the important part: it reads the frozen trace rather than calling a model, so the result is deterministic and the run measures the memory, not the model that happened to be available on the day.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_capstone.py --mode replay

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_capstone.py --mode replay

Python 3.11 or later. Replay mode makes no model call and needs no service. The run writes a new run directory under experiments/benchmark/runs/; a frozen run is never overwritten, and each run gets its own directory and manifest.