Part 2 — Get the Model Out of the Chat Box

The book you could write better next year

Write a book with AI this year and you will be able to write the same book better next year. And better again the year after. The same is true of the code, the research synthesis, the review process, the classifier — anything where a model does part of the work.

That is good news. It is also a statement about the ground you are building on, and most software is not built for ground that behaves like this.

Traditional software sits on a foundation that holds still. The database you chose in 2019 answers the same query the same way in 2026, and the compiler does not wake up one morning better at some programs and worse at others. You pour the foundation once and build upward.

The model underneath an AI system is not that kind of thing. GPT-3 and the models that followed it are not versions of one product the way database releases are. The model you build on at launch and the one available a few generations later are different machines, with different strengths and failure shapes, different prices, and their own reading of what your prompt meant. The next few years will bring at least as many releases, quite possibly with gains of similar size.

What do you build on, when the most important component underneath you will be replaced by something substantially different — repeatedly — for as long as the system exists?

The ground is moving, measurably

Three pieces of evidence, each describing a different way it moves.

Capability moves fast, in one direction. METR proposed a practical measure: the length of task, in human time, that a model can complete with 50% success. On their software and research task suites, the frontier’s 50% time horizon doubled roughly every seven months from 2019, possibly faster in 2024, with Claude 3.7 Sonnet at around fifty minutes when they published (Kwa et al., 2025). A seven-month doubling compounds to roughly ten times every two years. The authors are explicit that external validity is limited and that any extrapolation is conditional, so treat that as the trend’s arithmetic, not a forecast. Even heavily discounted, the gap between what you build on at launch and what is available two years later is not incremental.

The “same” model moves too, in no particular direction. Chen, Zaharia, and Zou compared the March and June 2023 versions of GPT-3.5 and GPT-4 — same product names — across seven task families. GPT-4’s accuracy at distinguishing prime from composite numbers fell from 84% to 51%, associated with a drop in how well it followed chain-of-thought prompting; code-generation formatting errors increased for both models; some other capabilities improved (Chen, Zaharia & Zou, 2023). Their conclusion is the one to keep: the behavior of the “same” LLM service can change substantially in a short time, so it needs continuous monitoring.

A small note that is itself on-theme. An earlier version of that paper reported a far more dramatic prime-number collapse, and that larger figure is still the one circulating. The authors revised it. Cite the current version.

Upgrades break specific things while improving the average. Echterhoff and colleagues studied what happens to task-specific adapters when the underlying base model is updated. They found negative flips — instances the old model got right that the new one gets wrong — across a diverse range of tasks and models, even when the downstream training procedure stayed identical. Their compatibility method cut negative flips by up to 40% when moving from Llama 1 to Llama 2 (Echterhoff et al., 2024). The method is not the point. The point is that “the new model scores higher” and “the new model still does the thing I depended on” are different claims, and the first does not imply the second.

So: capability compounds, named models drift, and upgrades regress on individual cases. None of those is a foundation.

Finish the frame, turn the chamber

Chapter 8 argued that projects are supposed to end. This chapter argues that AI work never does. Both are true, and the contradiction dissolves once you notice they describe different parts of the system.

Chapter 3 confined stochasticity to one box. Everything else — context assembly, the harness, the verifiers, the ledger, the policy, the router — went on the deterministic side. That split was argued on reliability grounds. It turns out to be what makes a system survivable over time.

You finish the frame. You keep turning the chamber.

The frame — the deterministic parts — is exactly the kind of software that gets done. The specification settles, the verifiers stop changing, and the adapter boundary absorbs a new provider in an afternoon. That is a finished product in Chapter 8’s sense, and it should be finished.

The chamber — the one stochastic slot — is the part that keeps changing, on the market’s schedule rather than yours. And because it is one slot with a defined interface, replacing what occupies it is a re-run, not a rewrite.

The frame is also the part you own. Your context rules, records, checks and policies stay yours while occupants come and go. That ownership is what lets a working environment built around you last, instead of being a bet on one vendor’s model — the argument Chapter 30 closes the book with.

Now look at the alternative. A system where stochastic work is spread everywhere — model calls parsing, deciding, routing, checking — has no frame to finish. Every release touches every part of it, and each generation becomes a rebuild. That is another answer to Chapter 8’s question about where the finished projects went: teams chasing model releases without a finished frame do not finish, because every release restarts them.

It also re-prices Chapter 3’s rule. Every operation you moved to the deterministic side is an operation that does not rot when the model changes.

The frame holds still while the chambers turn:

    flowchart TD
    subgraph FRAME["deterministic frame — finished"]
        direction TB
        SP["specification"]
        CX["context assembly"]
        VG["verifiers"]
        LP["ledger · policy · router"]
    end
    subgraph CH["chambers — one stochastic slot each"]
        direction TB
        C1[("fast-classify<br/><i>occupant, version, price, date</i>")]
        C2[("deep-review<br/><i>occupant, version, price, date</i>")]
    end
    FRAME --> CH
    CH -.->|"new release: re-run<br/>against history, not rewrite"| CH
  

The chambers

A revolver holds several rounds and you choose which one to fire. The useful version of that picture: your system has several chambers — logical roles — and each is occupied, at a given moment, by a specific model.

A chamber is named for its job, not its occupant: fast-classify, draft, deep-review, critic-a, critic-b. The occupant is recorded with its exact model id, version or snapshot, price, and the date it was loaded.

What you choose on, per chamber:

The last row decides swaps, and it is chamber-specific. A new model can win deep-review and lose fast-classify in the same week. A global “upgrade to the new model” decision throws that information away.

An illustrative swap shows what the decision looks like on paper — not a measured result, a worked example of the arithmetic. Suppose one chamber’s frozen set has 40 items. The incumbent passes 28, the candidate passes 31. Paired per item: 26 pass on both, 7 fail on both, 5 pass only on the candidate, 2 pass only on the incumbent.

The paired difference is +3 items (+7.5 points) with a standard error near 6.5 points, so the interval comfortably crosses zero — a higher average that could easily be chance, with 2 negative flips on items you may depend on. Price it: say the replay cost $0.40 on the incumbent and $1.20 on the candidate, so cost per passing item is about $0.014 against $0.039. The candidate wins the average and loses utility by nearly three to one.

That is not a swap; it is at most a note saying which five items improved and whether the two flips matter, written down with both occupant versions, the numbers, and who approved.

This is not an invented abstraction. CodeAI resolves a chamber name to its occupant through .codeai/config.toml, and the mapping is small enough to read at a glance. From one of CodeAI’s regression tests:

[models.deep-review]            # the chamber, named for its job
adapter  = "opencode"
model    = "mimo-v2.5"          # the current occupant
protocol = "chat_completions"   # how that occupant is served

ModelConfig.resolve_chamber("deep-review") returns that mapping, and nothing else in the system needs to know which model it names. Changing the occupant is an edit to configuration, not to code. The provider layer adds a versioned pricing table that returns no cost at all — never zero — for a model it does not recognize (PRICING_VERSION and estimate_cost_usd in providers.py). Chapter 28 finds the limit of that caution: the same table matches model names by prefix, so gpt-4o-mini was priced at gpt-4o rates. Refusing to invent a price for an unknown model is not the same as pricing a known one correctly.

That is the chamber mechanism in its smallest form: a name the rest of the system depends on, bound to an occupant that can change. The full release protocol below is never built in this book as a single mechanism. Its parts are: Part 5 builds paired comparisons under frozen decision rules, and Chapter 28 puts several occupants on the named rungs of an escalation ladder.

New rounds cost more; old rounds get cheap

A newly released frontier model usually costs more than what it replaces. Meanwhile the price of reaching a fixed level of capability has been collapsing — Epoch AI measured declines of 9× to 900× per year depending on the milestone (Epoch AI, 2025; see Chapter 6).

Put those together and the revolver has an economic logic:

• Load the newest, most expensive round only into chambers where it measurably clears a verifier that cheaper rounds do not.
• Everywhere else, last year’s frontier — now a fraction of its launch price — is often the right occupant, if it passes.
• Re-check chambers when prices change, not only when models are released. A price drop is a reason to re-run the harness too.

Keep that up and you accumulate something no vendor can sell you: a measured map of cost per unit of intelligence, per chamber, for your own work. That map is where the advantage lives.

A release is an experiment, not an upgrade

When a new model arrives, the tempting move is to try it on a few things you remember, like what you see, and switch. That is Chapter 7’s twenty-examples mistake, made at the moment of maximum enthusiasm.

The protocol instead, using what Part 1 already built:

1. Pick one chamber. Never evaluate “the new model” in the abstract.
2. Replay that chamber’s frozen task set against the candidate, at n ≥ 3 per item, with the candidate’s own best prompt rather than the incumbent’s.
3. Compare paired, per item, against the current occupant — not two summary scores. Report the difference with a standard error.
4. Count flips in both directions. Items the incumbent passed that the candidate fails, and the reverse. A higher average with flips on items you depend on is not a win for that chamber. A handful of flips can also be chance: if only four items differ and three favor the candidate, a fair coin produces a split at least that lopsided more often than not. And a win on a weak verifier is not a win. Fix the check before you trust the swap.
5. Keep the critic fixed. The critic is itself a chamber. Never swap the judge in the same step as the thing it is judging, or you cannot tell which change moved the score (Chapter 7).
6. Price the result: cost per passing item, incumbent versus candidate.
7. Decide per chamber and record the decision — date, both occupants with versions, the numbers, who approved — and keep the incumbent reachable for rollback.

That is perhaps a day of work per release, for each chamber that matters. It is also the difference between adopting models on reputation and adopting them on evidence.

And because named models drift, a scheduled re-run of each frozen set against the current occupants belongs in the same machinery. You still monitor when nothing was swapped.

Your history is a regression suite

Here durable memory from Chapter 9 pays off in a way the design chapter did not mention.

If the system kept its inputs — the compiled context packages, the specification in force, the verifiers, the raw outputs — then every piece of past work is two things at once.

It is a regression test for the next occupant: here is a task, here is what passed before, does the candidate still pass it?

And it is a regeneration queue. The book you wrote with this year’s models can be rewritten next year — not from memory, but by replaying the recorded tasks against a better chamber, and measuring per item whether the new result is actually better before anything replaces the old one.

Without preserved inputs, “we could do this better next year” means starting again from a blank page. With them, it means a replay and a measurement.

Two limits, both from Part 1. Regenerating everything is Chapter 6’s invoice in a new costume; regenerate where the measured gain is worth the price, not by reflex. And not everything should revolve. Work that has been distilled or made deterministic on Chapter 6’s ratchet is done, and a stable artifact that passes its verifier needs no new model at all. That is a feature, not a missed opportunity.

Where this breaks

• Trends end. The seven-month doubling is a historical fit with a stated external-validity caveat. Build for replacement because releases happen, not because a curve promises the next one will be better.
• Prompts are partly model-specific. A prompt tuned for one occupant can underperform on another, so a swap may need prompt work. The specification should record the prompt per chamber per occupant.
• Vendors set part of the schedule. Deprecations force swaps whether or not a candidate has been evaluated. The protocol above is how you avoid being forced into an untested one.
• Evaluation costs money every release. The harness is not free to run (Chapter 7), which is another reason to evaluate the chambers that matter rather than everything.
• Drift without a swap is still drift. A pinned model name does not guarantee pinned behavior.

Do this now

Twenty minutes. Load the cylinder.

1. List every place your system calls a model. Give each a chamber name for its job, not for the model in it.
2. For each chamber, record the current occupant: provider, exact model id, version or snapshot date, input and output price, and the date you last evaluated it.
3. Count how many places in your code a concrete model id is written directly rather than resolved from a logical name. Each one is a place where a release costs you a code change instead of a configuration change.
4. For your most important chamber, write down what it would take to replay its frozen task set against a candidate tomorrow: which tasks, which checks, which critic, and how you would compare per item.

If step 4 has no answer, that — not a model choice — is your next piece of work.

Failure modes

• Treating the model as a foundation. It will be replaced by something substantially different, repeatedly.
• Hard-coding model ids. Every release becomes a code change and every comparison becomes a branch.
• Upgrading globally. A new model can win one chamber and lose another; a single switch discards that.
• Adopting on aggregate score. Averages hide negative flips on the items you depend on.
• Swapping the critic and the generator together. You can no longer attribute the change.
• Assuming a fixed name means fixed behavior. Named models drift; monitor on a schedule.
• Spreading stochastic work through the frame. Every release then touches everything, and nothing finishes.
• Regenerating by reflex. Re-run where measured gain justifies cost; leave finished deterministic work alone.
• Not keeping inputs. Without them, “better next year” means starting over.

What this chapter established

• The model is not a foundation. Capability has compounded fast (a roughly seven-month doubling in task horizon since 2019, with the authors’ caveats), named models drift (GPT-4’s prime-identification accuracy fell from 84% to 51% between March and June 2023), and upgrades cause negative flips on individual cases even as averages rise.
• Chapter 8 and this chapter describe different parts of a system: finish the frame, turn the chamber. The deterministic frame is finishable; the one stochastic slot keeps changing, and because it is one slot, replacing its occupant is a re-run rather than a rewrite.
• Teams without a finished frame rebuild on every release — another reason projects do not finish.
• A system has chambers named by job, each with a recorded occupant, chosen per chamber on speed, quality, knowledge, applicability, utility, and gain over the incumbent. CodeAI’s logical model names and versioned pricing table are the smallest form of this.
• New frontier models cost more at launch while fixed capability gets cheaper fast. Load expensive rounds only where they clear a verifier cheaper ones cannot, and re-evaluate on price changes as well as releases.
• A release is an experiment: one chamber, replay its frozen set, compare paired per item, count negative flips, hold the critic fixed, price per passing item, decide and record per chamber, keep rollback.
• Preserved history is both a regression suite for the next occupant and a regeneration queue for better work — measured before anything is replaced.
• The durable advantage is not access to the newest model, which everyone has. It is a measured map of where each model earns its cost on your own work.

Next

Every mechanism in this chapter rests on one unglamorous fact being true at the lowest level: each call records exactly which model answered it. Which provider, which model id, which snapshot, at what price, after how many attempts, with what raw output.

If the call does not record that, chambers cannot be compared, negative flips cannot be counted, drift cannot be detected, and your history is not a regression suite — it is a pile of text with no provenance.

So construction starts there.

Continue with The Smallest Useful Model Call.

References

• Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. (METR). Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499, 2025. https://arxiv.org/abs/2503.14499
• Lingjiao Chen, Matei Zaharia, and James Zou. How Is ChatGPT’s Behavior Changing over Time? arXiv:2307.09009 (v3), 2023. https://arxiv.org/abs/2307.09009
• Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, and Hadi Pouransari. MUSCLE: A Model Update Strategy for Compatible LLM Evolution. Findings of the Association for Computational Linguistics: EMNLP 2024. https://arxiv.org/abs/2407.09435
• Epoch AI. LLM Inference Prices Have Fallen Rapidly but Unequally Across Tasks. Data Insight, March 12, 2025. https://epoch.ai/data-insights/llm-inference-price-trends

Implementation sources: CodeAI, src/codeai/modelconfig.py (ModelMapping, ModelConfig.resolve, load_model_config) and src/codeai/providers.py (PRICING_VERSION, PRICING_TABLE, estimate_cost_usd). Inspected, not yet exercised in a chamber-swap walkthrough.