Part 5 — More Intelligence Is Not Automatically Better

Seven repairs, one unresolved bug

Open on the stubborn task, before any percentage. The problem statement asks for a function that returns the mean of the list it is given, on every call. The starter code caches its first answer and never lets go:

def average(xs):
    if not hasattr(average, '_n'):
        average._n = len(xs)
        average._s = sum(xs)
    return average._s / average._n

The reference repair is one line: return sum(xs) / len(xs). Delete the state.

Seven candidate repairs were generated across three arms, and none deleted the state. Reading them from the frozen rows: ¹

The draws varied, the models varied, and so did the code, but one idea ran through all seven: the state belongs to the function and needs fixing rather than removing. That is the chapter’s question in concrete form. Did changing model identity expose a solution that repeated baseline sampling did not? On this corpus, no.

Did changing models add verified coverage beyond repeated sampling?

More names, more candidates, more coverage

The distinction is more model names ≠ more candidates ≠ more coverage. Count solved tasks, not models. Two denominators must never be substituted for one another:

In P1 the C1 and H1 arms happen to score 0.9167 on both — 33 of 36 candidates, 11 of 12 tasks — and that coincidence is itself a trap. The fractions share a value and answer different questions.

A third boundary rides alongside: the oracle is not a deployable selector. When an arm generates three candidates and one passes, oracle@3 counts the task as covered because the experiment’s checker identifies the passing candidate after the fact. That does not show that a production system could pick it without the checker. Chapter 25 needs this boundary badly; here it stays attached to every coverage sentence. ²

Constructed teaching fragment, executed as arithmetic, not as evidence. Why the denominators diverge in general, even when they coincide in P1: ³

candidates = {
    "task-a": [True, True, False],
    "task-b": [False, False, True],
    "task-c": [False, False, False],
}
candidate_passes = sum(p for draws in candidates.values() for p in draws)   # 3
task_coverage = sum(any(draws) for draws in candidates.values())            # 2
# 3/9 candidates pass, but 2/3 tasks are covered.

The experiment design

The comparison is a matched triple. C0 is one draw from the baseline model. C1 is three draws from the same baseline model. H1 is one draw each from three models, the baseline plus two others. The comparison that matters is H1 against C1, because candidate count is matched: C1 asks what three same-model draws buy, and H1 asks what model variety buys on top of that. ⁴

Interpretation was preregistered before any model call. Rule 2 covers what happened: if the portfolio roughly ties the homogeneous arm, keep repeated sampling as the simpler default and look for task-specific unique rescues. Rule 5 covers the stubborn task: if a task defeats every arm, investigate framing or generator capability before collaboration machinery. ⁵

Two parts of the preregistered design did not survive contact with the environment, and both change what P1 can say. The preregistration named H1 as the baseline model plus Claude and GPT. Cloud keys were unavailable, so the run used two local models instead: mistral:7b-instruct and llama3.1:8b. P1 therefore tested variety among three small local models, not the cross-vendor variety the design anticipated. And the companion money-matched framing was never realizable, because local models have no measured monetary cost; token and latency budgets governed instead. ^{5 4}

What CodeAI does

The corpus is seeded-code-v1: twelve seeded repair tasks, each with starter code, a reference solution, and hidden tests the model never sees. Arms run through sealed fan-out, each candidate is materialized in a workspace, and each hidden-verifier outcome is recorded per candidate. ^{6 1}

The metric definitions live in one place, and the recomputation reuses them. Oracle coverage is the share of tasks with at least one passing candidate. Candidate rate is passes over generated candidates. A rescue is a set difference: tasks solved by one arm and by no other. Conditional failure is P(B fails | A fails) over jointly attempted tasks. The heterogeneity premium is H1’s coverage minus C1’s. The report carries its own warnings: fewer than 30 tasks forbids causal-superiority inference, and oracle@k is best-of-k with an oracle selector, not deployable performance. ⁷

No runtime change accompanies this chapter; the recomputation equals the historical report exactly.

Two outside results explain the design without contributing evidence to it. Sampling one model repeatedly is the baseline any heterogeneous portfolio has to beat; self-consistency shows how much repeated sampling of one model can buy on its own (Wang et al., 2023). And classifier-ensemble research has long found that diversity among members does not straightforwardly produce ensemble accuracy — useful differences have to be measured, not inferred from distinct labels (Kuncheva, 2003). That is why P1 matched draws first and counted rescue sets rather than model names.

The frozen run, reconstructed

Eighty-four calls over twelve tasks — 12 baseline draws, 36 homogeneous redraws, 36 portfolio draws — produced 84 candidates and 83 checks: mistral’s syntax error never reached the hidden tests. The exports are byte-pinned with verified hashes, the analysis recomputes arm tables from candidate and call rows, and an independent verifier asserts the figures and catches a seeded corruption. ^{2 1}

The three arms solved the identical eleven tasks. The failed-task set is the same single task everywhere, which is why every pairwise conditional failure reads 1.0: whenever one arm failed a task, the other failed it too — because there was only one task to fail. That number describes twelve tasks. It is not evidence that model errors are generally correlated, and the chapter’s title must not do that work.

The candidate rows make the ceiling even flatter than the table suggests. On the eleven easy tasks, all 77 candidates passed — every draw, from every model, in every arm. On the twelfth, all seven failed. There is no per-draw variation anywhere in this run for a portfolio to exploit, in either direction. ¹

Unique rescues in either direction: none. Premium: 0.0. ²

Resources differed while outcomes did not. Tokens ran 1,827 for C0, 5,452 for C1, and 6,549 for H1 — the portfolio at 1.20× the matched homogeneous arm. Reported median call latency was 2.9 s, 2.8 s, and 6.8 s. Tokens, latency, and money stay separate columns: money was never measured, and a median per-call latency is not portfolio wall-clock time. The supported cost sentence is only that H1 spent more tokens for the same verified coverage. ²

The run in one compact table, read from the frozen report:

Heterogeneity premium (H1 − C1): 0.0. Coverage is identical down the column while tokens rise across it — the table is the result. ²

The ceiling

C0 solved eleven of twelve tasks with a single draw. That fact dominates the run. With exactly one task of headroom, the portfolio gained none:

baseline coverage     = 11/12
maximum possible gain = 1 task
observed gain         = 0

P1 still stands as a measurement. A null under a ceiling says this corpus cannot discriminate between the interventions, not that they are indistinguishable in general. Hence the honest response is a harder corpus, not a stronger claim, which is why Chapter 25 exists.

In the checker the same ceiling appears. The hidden test for the stubborn task begins with assert average([1, 2, 3]) == 2 or True, an assertion that cannot fail; it exists only to seed stale state. The two assertions after a module reload do the discriminating. That is fine for this task — the reload assertions are sharp — but it is a reminder from Chapter 21 that a check’s adequacy is a property of each assertion, not of the file. ⁸

An earlier plan for this book imagined a “Council” — reviewer, editor, brand guardian — improving a tutorial through complementary perspectives. That was a belief about variety, not a measured improvement, and it described prompted roles rather than independent sources. P1 replaces the assumption with a checkable question and returns a bounded answer: on this corpus, variety of labels bought nothing. ⁴

Checking it without trusting it

The analysis verifier asserts the recomputed coverage, candidate, and token figures per arm, requires zero premium with zero rescues for P1, and must reject a mutated copy. The export verifier checks all four frozen bundles’ hashes. The candidate table in the opening and the 77-of-77 ceiling count were read from the frozen rows for this chapter; they are outside both verifiers. ^{2 1}

The limits travel with the numbers. Coverage is not selection. Twelve tasks forbid causal or significance claims, and the machinery emits its own tiny-n warning. Three small local models on one seeded corpus say nothing about other families, sizes, or vendors — and these three may share training data and failure modes that P1 never measured, so the chapter explains nothing about why the overlap is complete. ^{2 4}

What this is not

• Not proof that heterogeneity never helps. Twelve tasks, three local models, one task of headroom.
• Not the experiment that was preregistered. Claude and GPT were replaced by local models; cross-vendor variety was not tested.
• Not equivalence of models. Same coverage establishes the same verified set, not equal ability.
• Not a collaboration finding. No debate or synthesis was tested; Rule 6 forbids conclusions about it.
• Not an economic comparison. No money was measured.

Where it is still weak

1. Ceiling-bound. One task of possible gain, and no per-draw variation on the other eleven. ¹
2. Tiny-n. Twelve tasks forbid causal and significance claims. ⁹
3. Substituted portfolio. Local mistral and llama stood in for the preregistered cloud models. ⁴
4. Single seeded corpus. Synthetic repair tasks need not resemble any real workload. ¹
5. Unexamined causes. Overlap in training data or prompting behavior was never measured.
6. Oracle, not selector. Coverage counts a task when any candidate passes; nothing chooses among candidates.

Do this now

Thirty minutes. Count solved tasks, not model names.

1. Take any multi-candidate result — three drafts, three reviews, three runs. Mark each candidate pass or fail with a check you trust, and compute both denominators.
2. Build the task-by-arm table. Tasks where every candidate passes are your ceiling; they teach nothing about which arm is better.
3. For the tasks every arm failed, read the failed candidates side by side, as the opening table does. Name the shared idea they all got wrong.
4. Compare the portfolio you ran with the one you planned. If anything was substituted, write it into the result.
5. Write the decision in one sentence with the corpus attached: “keep the simpler default on this workload until this harder set shows otherwise.”

If you are building with an assistant:

Measure variety by solved tasks, never by model names. Match candidate
counts before comparing portfolios: one draw, N same-model draws, N mixed
draws. Keep candidate pass rate and task coverage separate, report the
failed-task intersection and read the failed candidates side by side.
Preregister the interpretation, including what a tie means, and record any
substitution of the planned models. Treat the oracle as a retrospective
finder, not a selector. Keep tokens, latency, and money in separate
columns. When the baseline covers nearly everything, name the ceiling and
build a harder corpus.

Failure modes

• Counting names instead of tasks. Three models sounds like more than one until the solved sets come back identical.
• Comparing H1 to C0. The portfolio must beat matched same-model sampling, not a single draw.
• Quoting 33/36 as coverage. Candidate passes repeat on solved tasks.
• Reading 1.0 overlap as law. A complete intersection on one failed task describes the corpus.
• Reporting the planned portfolio. If the models changed, so did the question.
• Arguing with the ceiling. A null under 11/12 baseline coverage indicts the corpus’s power, not the experiment’s honesty.

What this chapter established

• H1 matched C1 exactly: 33/36 candidates, the same 11/12 tasks, zero rescues, premium 0.0, at 1.20× the tokens. ²
• All 77 candidates on the eleven easy tasks passed and all 7 on the stale-state task failed, each by repairing the state instead of removing it — so the run had no variation for a portfolio to use. ¹
• The portfolio tested was three local models, substituted for the preregistered cross-vendor set. ⁴
• The preregistered Rule 2 keeps repeated same-model sampling as the simpler default until a harder corpus shows otherwise. ⁵

Next

The baseline keeps its place on easy ground, and the measurement says exactly that. The response to a ceiling is a harder test: problems difficult enough that methods have room to differ, where a portfolio might cover more while producing worse candidates.

Continue with Make the Problems Harder.

References

• L. I. Kuncheva. That Elusive Diversity in Classifier Ensembles. Proc. IbPRIA 2003, LNCS 2652, pp. 1126–1138. Publication record.
• Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023. arXiv:2203.11171.

Implementation sources: P1 ran on the P-series CodeAI lineage (seeded-code-v1 corpus; src/codeai/corpus.py: CORPUS_VERSION, seeded_corpus; src/codeai/experiments.py: ArmDef, run_arm; src/codeai/analysis.py: arm_metrics, conditional_failure, unique_rescues, build_report). No CodeAI change accompanies this chapter. The candidate table and ceiling count were read from frozen rows during editing; the starter, reference, and hidden test come from the corpus source. Evidence: experiments/applied-ai/evidence/p-series/ (frozen exports, hashes verified), experiments/applied-ai/evidence/p-series-analysis/ (analysis.json, TABLES.md, analyze_pseries.py, verify_analysis.py), C:/Projects/codeai/experiments/P1-results.md and P1-interpretation.md. Nothing was rerun and nothing frozen was modified.