Staged gates beat one big prompt, and the ablation shows it

A derived loop is a consequence nobody wrote down: a change in one record that implies something about another. The chapter’s claim is that these are hard enough that the order of the gates matters, and the suite tests that by ablation: the same derived loops, with one stage removed at a time.

Run it

git clone https://github.com/ernanhughes/project-memory.git
cd project-memory
git checkout 0a6419ab0d6f65e491374b971d214255f2b0d8e8
python solution/run_ch11.py --outdir ./runs/ch11-demo

Python 3.11 or later. The suite is deterministic with zero model calls. It writes a new run directory; frozen runs are never overwritten.

Expected outcome

Seven policies, and the two numbers that decide the question:

  unconstrained  prec=0.4583 rec=1.0000 micro=0.4762/1.0000 harm_tasks=4 abst=0.0000
  staged         prec=1.0000 rec=1.0000 micro=1.0000/1.0000 harm_tasks=0 abst=1.0000
  no-current     prec=0.9286 rec=1.0000 micro=0.9091/1.0000 harm_tasks=0 abst=1.0000
  no-expected    prec=0.8571 rec=1.0000 micro=0.8333/1.0000 harm_tasks=0 abst=1.0000
  no-cancel      prec=0.6458 rec=1.0000 micro=0.5556/1.0000 harm_tasks=4 abst=0.0000
  oracle         prec=1.0000 rec=1.0000 micro=1.0000/1.0000 harm_tasks=0 abst=1.0000
  abstain        prec=null rec=0.0000 micro=null/0.0000 harm_tasks=0 abst=1.0000

The unconstrained policy and the one with cancellation removed both harm four tasks and abstain on none. The staged policy reaches the oracle’s precision with perfect recall and no harm.

The backtest is the part that says what the gates are for:

  backtest v2-drop-cancelling-leg: gain=-0.3542 promoted=False

Removing the cancelling leg makes a proposed change worse, and the backtest refuses to promote it. The gate is not decoration: it blocked a change that would have lost ground.

Reading the rows

The abstain row has prec=null. That is not a bug in the row and not a zero: an abstaining system makes no claims, so its precision is undefined while its harm count is 0. The micro column reports precision and recall together for exactly that reason, so a reader can see the undefined case rather than having it silently rounded to zero.

Mechanism and limitations

Documented: the policies, the metric names and the backtest rule are in solution/run_ch11.py and the solution/derived_loops/ module. The chapter’s own text names the frozen run and the policy version, and says the run has zero model calls.

Observed: the run above. The staged policy matched the oracle on precision and recall, harmed no task, and abstained on every task. Removing the current state, the expected state or the cancellation each cost precision; removing cancellation also restored the harm. The backtest refused to promote the change that dropped the cancelling leg.

Not shown. This is an ablation on one fixture with zero model calls: it shows which gates matter for this set of derived loops, not that the staged policy would be better on real traffic. The oracle row is a ceiling, not an achievable result. Precision here is precision over a fixture’s labels, and the fixture’s labels are the harness’s. Nothing here measures whether a person acted on a derived loop, or whether a loop they were told about was one they would have recognised.

Canonical source and the derived-loop suite .