Part 5 — More Intelligence Is Not Automatically Better

Three tasks the baseline never solved

Chapter 24 ended under a ceiling: eleven of twelve tasks solved by a single draw, one shared failure, nowhere for a portfolio to go. The response was a harder corpus: forty repair tasks in four strata of ten, each stated as misbehavior plus a contract rather than a fault label, with hidden tests defining correctness. On that ground the arms finally differ. Lead with the paired difference, not the percentages: ^{1 2}

H1-only solves (3):  v2-strip-query, v2-half-up-rounding, v2-keyed-cache-length
C1-only solves (1):  v2-splitlines-cr
net portfolio gain:  +2 tasks

Everything else in the chapter is commentary on those four rows. One piece of commentary comes first, because it sets the volume for the rest: four discordant tasks splitting three to one is what a fair coin produces often. An exact two-sided sign test on 3 against 1 gives p = 0.625. The arms differ on this corpus; the difference is not evidence that one arm is better. ³

Can a portfolio cover more tasks while producing worse candidates?

Potential coverage, delivered reliability

The distinction is potential coverage ≠ delivered reliability, and inside it a second one: a rescue ≠ a model-specific rescue. A solve by another draw of the same model is sampling variation. A solve by a different model, where every baseline draw failed, is the shape complementary coverage is supposed to have.

Read both denominators at once: ¹

The portfolio covered two more tasks while producing fourteen fewer passing candidates in the same 120 slots. That is not a contradiction, and the frozen rows show exactly why.

Why the inversion happens

H1 is not “three models” in the abstract. Per task, it spends one slot on the baseline model and two slots on other models. Splitting H1’s candidates by the model that produced them: ³

The candidate rate fell because two of every three slots went to models that pass far less often than the baseline. All three of the run’s compile-gate execution errors came from one of them. Coverage still rose, because coverage asks only whether any slot passed, and the weaker models occasionally passed where the baseline did not. More needles, in more hay.

Two details in that table keep the story honest. First, H1’s single qwen draw per task covered 30 tasks — more than C0’s single qwen draw (27) — so draw-to-draw variation in the same model moves coverage by several tasks on this corpus. Second, the aggregate hides task-level exceptions: H1 passed more candidates than C1 on six tasks. “The portfolio is weaker per candidate” is true in total, not everywhere. ³

Reliability asks how often candidates are useful; coverage asks whether at least one is. Neither says whether a real system could find the useful one. Without a selector, the portfolio’s extra coverage is potential, not delivered. Generate-then-select architectures recover performance only by adding a separate component that ranks candidates (Cobbe et al., 2021) — which is also why their result cannot be borrowed as evidence that P1.1’s candidates were selectable.

Four exceptions, four different stories

Each paired exception needs its own reading, because the aggregates mislead about all four. The model identities and raw outputs below are frozen row facts; the classifications follow the historical report, including its uncertainty. ^{3 4}

v2-strip-query: the clean model-specific rescue. The task needs a URL returned without its query component. All four baseline-family draws failed — C0’s one and C1’s three — and so did H1’s own baseline draw and its llama draw. The mistral draw passed, and it passed by a different route: instead of string surgery, it parsed the URL with urllib.parse and rebuilt it with the query replaced by an empty string. Baseline draws fail, a different model succeeds by a different approach. This is the shape a diversity rescue is supposed to have, observed once. ³

v2-half-up-rounding: sampling variation, not diversity. The task needs halves rounded away from zero. Every C0 and C1 draw failed. The passing H1 candidate came from H1’s baseline-model slot:

def round_half_up(x):
    return int(x + 0.5) if x >= 0 else int(x - 0.5)

Mistral and llama both failed. Same model as the baseline, different draw, different outcome. Calling this a diversity win would credit model identity for what the rows show as variation within one model. ³

v2-keyed-cache-length: ambiguous, kept ambiguous. All three H1 draws passed; all four baseline-family draws failed. That pattern fits two stories. The portfolio may genuinely help on this task — two other models passed independently. Or the baseline model’s H1 draw was simply lucky where its four twins were not, and the other models happen to handle the task well. The rows cannot separate those, and the report declines to choose. So does this chapter. ^{3 4}

v2-splitlines-cr: the portfolio’s loss. One of C1’s three draws solved the carriage-return splitting task; all three H1 draws failed it. The portfolio did not only add coverage; it also missed a task the baseline found. Any account of the +2 that omits this −1 is advocacy. ³

Two more row facts bound the story. C0’s single draw solved two tasks, v2-lsp-square and v2-unsound-cache, that all six C1 and H1 draws missed: one draw beating six is sampling variation cutting the other way. And four tasks — registry pollution, retry-once-accepted, size format, stable priority — defeated every arm. They are the new hard core and the next ceiling. ¹

So of three H1-only solves, exactly one carries the model-specific rescue shape.

Where the gain sits, carefully

Coverage by stratum, recomputed from frozen rows using the stratum labels in the semantic-repair-v1 corpus definition, and matching the P1.1 report’s table: ^{3 2}

Read at face value, the portfolio’s edge lives in local and boundary tasks. Read with the sample size, each of those edges is one task out of ten, and the architectural row shows a single baseline draw beating its own triplet. The report flags that as sampling noise and a caution against over-reading. Ten tasks per stratum cannot support a story about which kind of bug “needs diversity,” and the chapter does not tell one. ⁴

“Harder” needs the same care. semantic-repair-v1 was harder for this setup: single-draw baseline coverage fell from 11/12 to 27/40, which made room for the arms to differ. Nothing here establishes these as hard software-engineering problems in general, and the hidden tests define correctness narrowly per task. Test strength and task difficulty are separate axes: augmenting a code benchmark’s tests has been shown to cut measured pass rates substantially and even change model rankings (Liu et al., 2023).

What CodeAI does

The machinery is unchanged from Chapter 24: the corpus builder defines forty symptom-style tasks with strata, families, fault classes, and hidden tests, while the same three-arm protocol runs sealed fan-out per task and records a hidden-verifier outcome per candidate. Metric definitions for coverage, candidate rates, set-difference rescues, conditional failure, and the premium are computed from the rows in the same way. No CodeAI change accompanies this chapter. ^{2 5}

The run produced 280 calls, 280 candidates, and 277 checks: the three mistral execution errors failed a compile gate and never reached hidden tests. ³

The decision rule holds

The preregistered rules were written for P1 and bind P1.1 too. Rule 1 pursues heterogeneity only if the portfolio “materially beats” homogeneous sampling at similar cost. Rule 2 says that if they roughly tie, repeated sampling stays the simpler default. ⁶

“Materially” was never a number, and that is a real weakness of the rule. The P1.1 report judged +0.05 coverage — two tasks net, from a 3-to-1 discordant split, at about 30% more tokens and 2.5× median call latency, with no selector — to be a weak positive rather than a material win. So Rule 2 still governs, and homogeneous sampling remains the default. That judgment is the report’s, applied to a qualitative threshold; the sign test above supports it without having been part of the rule. ^{4 3}

The report also applies Rule 5 — investigate decomposition, context, framing, or stronger generators before adding collaboration machinery — to the four-task hard core. Rule 5 was written for arms that are poor overall; applying it to the subset every arm failed is a reasonable extension, and it is the report’s. ⁴

Resource language stays descriptive. There was no selector, no value per solved task, and no pricing for local models, so the two tasks carry no dollar figure and no cost-effectiveness verdict. ¹

Checking it without trusting it

The analysis verifier asserts every P1.1 arm figure, the +0.05 premium, and the exact rescue sets against the recomputation, and catches its seeded corruption; the export verifier checks all four frozen bundle hashes. ¹

The per-model split, the stratum table, the task-level pass comparison, and the sign test were recomputed from the same frozen rows for this chapter. They are not part of either verifier. The stratum labels come from the corpus definition in CodeAI source at corpus version semantic-repair-v1; that they reproduce the report’s table exactly is the check that the labels did not drift. ³

What this is not

• Not a portfolio advantage. Two net tasks from a 3-to-1 split is compatible with chance.
• Unproven specialization. One mistral rescue and one ambiguous case do not show that models own task types.
• Coverage without delivery. Coverage says a passing candidate existed; nothing here finds it.
• No general difficulty ranking. Harder for this setup, under these hidden tests.

Where it is still weak

1. Attribution is partly interpretation. Row identities and raw outputs are measured; the luck/rescue/ambiguous labels are the report’s reading. ⁴
2. Small and local. Forty tasks, ten per stratum, three local models; causes of overlap unmeasured. ¹
3. No selector. The +2 exists only under retrospective identification.
4. The threshold is qualitative. “Materially beats” was never a number. ⁶
5. Portfolio composition is one choice. Two weaker models in two of three slots; a different mix would give a different inversion. ³

Do this now

Thirty minutes. Separate your coverage from your reliability.

1. Take any result where you generated several candidates per task. Compute task coverage and candidate pass rate separately, then list the tasks exactly one approach solved.
2. Count the discordant tasks both ways and run a sign test before calling the difference a gain.
3. Split each portfolio’s candidates by generator. If the pass rate fell, find which slot pulled it down.
4. For each one-approach solve, classify it: different-model success, same-model redraw, or ambiguous. Leave ambiguous rows ambiguous.
5. Write the default decision with a numeric threshold attached: what gain, at what resource multiple, would change your process?

If you are building with an assistant:

Make problems hard enough that methods can differ, then match candidate
counts and lead with paired task differences before any aggregate: whose
solves, whose misses, net, and a sign test on the discordant tasks. Split
portfolio candidates by generator to explain any change in pass rate.
Classify each rescue from row identities: different-model success,
same-model redraw, or ambiguous. Keep coverage, candidate reliability, and
deployable selection as separate columns, report resources without pricing
them, and write numeric decision thresholds before the run.

Failure modes

• Leading with 82.5% against 77.5%. Aggregates hide the four rows that are the finding.
• Calling a 3-to-1 split a win. Four discordant tasks cannot carry that weight.
• Calling all three H1-only solves diversity rescues. One is a baseline-model draw; one refuses classification.
• Forgetting the C1-only solve. Net the gain or drop the claim.
• Blaming “portfolios” for the pass rate. The rate fell because of which generators filled the slots.
• Leaving “materially” undefined. A qualitative threshold turns every close result into a judgment call.

What this chapter established

• On a harder corpus the arms differed: H1 33/40 tasks, C1 31/40, from three H1-only and one C1-only solve — a net +2 that an exact sign test cannot distinguish from chance (p = 0.625). ³
• Coverage rose while candidate passes fell (64/120 against 78/120), because two of H1’s three slots went to models passing 0.475 and 0.375 of the time. ³
• Of three H1-only solves, one is a clean model-specific rescue, one came from the baseline model’s own draw, and one stays ambiguous. ^{3 4}
• The homogeneous default stands under the preregistered rules; the “material” threshold was qualitative. ⁴

Next

Coverage rose while reliability fell, and only one rescue in four carries the model-specific signature. The extra model identities bought two tasks at thirty percent more tokens, through slots that mostly failed. The next question drops model identity entirely: can cheaper variation — different wordings to one model — do the same work, and how do we tell an exploratory signal from something worth promoting?

Continue with Diversity Without More Models.

References

• Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021. Paper.
• Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210, 2023. Paper.

Implementation sources: P1.1 ran on the P-series CodeAI lineage (semantic-repair-v1 corpus; src/codeai/corpus_v2.py: semantic_corpus, CORPUS2_VERSION; src/codeai/experiments.py: ArmDef, run_arm; src/codeai/analysis.py: arm_metrics, unique_rescues, build_report). No CodeAI change accompanies this chapter. The per-model split, stratum table, task-level comparison, and sign test were recomputed from frozen rows during editing. Evidence: experiments/applied-ai/evidence/p-series/ (frozen exports, hashes verified), experiments/applied-ai/evidence/p-series-analysis/ (analysis.json, TABLES.md, analyze_pseries.py, verify_analysis.py), C:/Projects/codeai/experiments/P1.1-results.md and P1-interpretation.md. Nothing was rerun and nothing frozen was modified.