Score three memory systems against the same six tasks

A memory book needs an instrument before it needs a memory. The chapter’s claim is that the measurement has to be separate from the thing being measured, and that a number without a failure attribution is not a result. This is that instrument: six canned tasks, three canned systems, and a scorecard that says which failures belong to which cause.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
cd experiments/benchmark
python -B -m memory_measurement.demo

Python 3.11 or later. The demo has no dependencies, calls no model, and needs no service. It scores canned responses against canned labels, so it is a test of the instrument, not of any memory.

Expected outcome

The good memory scores cleanly, and the metrics it cannot measure are reported as pending rather than as zero:

System:    good-memory
Condition: canned-demo
Corpus:    demo-v0.1
Tasks:     demo-tasks-v0.1
Run:       demo-good-memory (manifest cb5905e3ee76652f)

      1.00  abstention_correctness
      1.00  current_state_accuracy
      1.00  decision_exactness
      1.00  historical_state_accuracy
      1.00  source_precision
      1.00  source_recall
      1.00  supersession_correctness
      0.00  unsupported_source_rate  (lower is better)

   pending  ranking.MRR
   pending  ranking.nDCG

The stale memory fails in a characterised way, and the attribution says which failure belongs to which cause:

System:    stale-memory
...
      0.83  abstention_correctness
      0.00  current_state_accuracy
      0.00  decision_exactness
      1.00  historical_state_accuracy
      0.50  source_precision
      0.42  source_recall
      0.00  supersession_correctness
      0.17  unsupported_source_rate  (lower is better)

Failure attribution
  demo-A:CONTEXT_OMISSION
  demo-B:CONTEXT_DISTRACTION
  demo-B:CONTEXT_OMISSION
  demo-B:MISSED_SUPERSESSION
  demo-C:CONTEXT_DISTRACTION
  demo-C:CONTEXT_OMISSION
  demo-C:CORRECT_EVIDENCE_WRONG_INTERPRETATION
  demo-E:CONTEXT_DISTRACTION
  demo-E:CONTEXT_OMISSION
  demo-E:MISSED_SUPERSESSION
  demo-E:STALE_STATE
  demo-F:FAILED_ABSTENTION
  demo-F:UNSUPPORTED_CLAIM

A stale memory and a fluent-wrong memory score differently, and the attribution is what tells them apart: one is missing, the other is wrong.

The pending rows matter

The scorecard separates three states, not two. A metric that was not measured is pending, and it is not silently zero. That distinction is the difference between “this memory system scored 0.42 on source recall” and “this memory system was never measured on source recall”, and collapsing them is how a benchmark becomes a marketing claim.

Mechanism and limitations

Documented: the instrument, the metric names, the attribution vocabulary and the frozen-run manifest are in the project-memory repository under experiments/benchmark/memory_measurement/. The chapter’s own text says this demonstration is described in the repository’s benchmark notes and that nothing in the section is a book result.

Observed: the run above. Three systems, the same six tasks, and a different failure signature for each. The good memory scored 1.00 on the eight metrics that were measured and left thirteen pending.

Not shown. The tasks are canned: the demo checks a scorer against known answers, so it says nothing about whether any of these scores would be reproduced on real traffic. A “good” system here is a fixture, not a deployed memory. The attribution labels are the harness’s categories, and assigning a cause is a judgement made by the harness, not a fact recovered from the system. Nothing here measures a reader, and the pending rows are pending for a reason: this instrument does not cover ranking, provenance, temporal validity, epistemic promotion, behaviour or cost.

Canonical source and the measurement harness .