Behavioural Evaluation on Programmer.ie: Modern AI programming

What Remembering Means

Mon, 14 Sep 2026 05:00:01 +0100

Consider a small software team and the assistant that sits alongside it. In June the team spends a week arguing about where to keep its event log. Someone tries keeping it in SQLite. The prototype works, then slows down under concurrent writes. The team decides to move the event store to PostgreSQL, recording the decision as adr-007 on 11 July. By August nobody thinks about this any more; it is just how the system works.

Does Better Context Change Behaviour?

Mon, 14 Sep 2026 05:00:12 +0100

Chapter 1 defined memory behaviourally: remove the past, rerun the present task, and ask whether behaviour changed and whether the change was an improvement. Eleven chapters later, that test has never actually been run. Chapters 3 to 11 built a pipeline — retrieval, structure, association, routing, lineage, temporal state, open loops, frames, derived consequences — and measured each stage against ledgers of what the context contains. During development, a later assembly experiment, reported in Chapter 14, exposed the measurement anomaly that motivates the behavioural experiment reported here: at a 768-token budget the assembled context holds required-evidence recall of only 0.72 yet the reader answers at 0.955 key-claim coverage, better than the full raw context at 0.879. Either the ledger overstates what the task needs, or the reader answers without the evidence the book calls required, or the metric gives false confidence. Answer quality alone cannot distinguish those cases. This chapter builds the instrument that can.

The Remembering System

Mon, 14 Sep 2026 05:00:17 +0100

The book began with a deliberately strict definition:

Memory is when retained past experience changes what the system does now.

Everything else — vector stores, graphs, temporal state, context selection, consolidation, procedures — is mechanism.

That distinction changes what a final chapter should do.

The original plan saved the behavioural test for the twentieth chapter and imagined a full stack in front of it: events, claims, provenance, belief, open loops, policy, context assembly, consolidation, compression, forgetting, outcome adaptation, procedures, then behaviour. The experiments did not follow that plan. Selection moved earlier. Context assembly was measured before long-term compression. Derived state produced both benefits and hazards. Simple mechanisms repeatedly matched or beat ambitious ones. Chapter 12 was moved forward specifically to test the behavioural definition before the book accumulated more machinery.