Example

Group the trials by a signature, not by the raw text

Run a seeded twenty-draw series, group the outputs by the chapter's behavioural signature, and read the shape raw text hides.

One draw from a mixed distribution carries almost no information about the distribution, so the debugging object is the answer-distribution and not the answer. It is read from a fixed projection of each output, and that projection has to be chosen before the trials or the histogram is a Rorschach test. This program composes the chapter’s Signature dataclass and its behavioral_signature projection, then runs a seeded twenty-draw series at temperature 0.7 and a ten-draw probe at temperature 0. The draws come from a simulation, not a model.

Run it

Save this as behavioural_signature.py and run it with Python 3.11 or later. It uses the standard library only.

"""Chapter 21: group a trial series by behavioural signature, not by raw text."""

import math
import random
from dataclasses import dataclass
from types import SimpleNamespace


@dataclass(frozen=True)
class Signature:
    schema_valid: bool
    supported_claims: int
    unsupported_claims: int
    action: str | None
    task_passed: bool


def behavioral_signature(result) -> Signature:
    return Signature(
        schema_valid=result.schema_valid,
        supported_claims=result.supported_claims,
        unsupported_claims=result.unsupported_claims,
        action=result.action,
        task_passed=result.task_passed,
    )


def draw(seed: int, temperature: float):
    """One simulated generation: it cites 4.2, or it falls back to general policy."""
    rng = random.Random(seed)
    cites = rng.random() < (1.0 if temperature == 0 else 0.6)
    return SimpleNamespace(
        text=f"amount 4.2 or policy (run {seed})",
        schema_valid=True,
        supported_claims=2 if cites else 1,
        unsupported_claims=0 if cites else 1,
        action="approve_4.2" if cites else "general_policy",
        task_passed=cites,
    )


def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k estimator (Chen et al., 2021). A selection ceiling."""
    if n - c < k:
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)


def report(label: str, results: list) -> None:
    groups: dict[Signature, int] = {}
    for r in results:
        sig = behavioral_signature(r)
        groups[sig] = groups.get(sig, 0) + 1
    passed = sum(r.task_passed for r in results)
    print(f"{label}  N={len(results)}")
    print(f"  distinct raw texts: {len({r.text for r in results})}")
    print(f"  distinct signatures: {len(groups)}")
    for sig, count in sorted(groups.items(), key=lambda kv: -kv[1]):
        print(f"  {count:>3} x task_passed={sig.task_passed} action={sig.action}")
    print(f"  pass@1: {passed}/{len(results)}")
    print(f"  pass@10 ceiling (unbiased, N={len(results)}): "
          f"{pass_at_k(len(results), passed, 10):.3f}")


report("baseline (temperature 0.7)", [draw(s, 0.7) for s in range(20)])
report("deterministic probe (temperature 0)", [draw(s, 0.0) for s in range(10)])

Expect twenty distinct raw texts collapsing to two signatures, and the probe collapsing to one.

Expected outcome

baseline (temperature 0.7)  N=20
  distinct raw texts: 20
  distinct signatures: 2
   14 x task_passed=True action=approve_4.2
    6 x task_passed=False action=general_policy
  pass@1: 14/20
  pass@10 ceiling (unbiased, N=20): 1.000
deterministic probe (temperature 0)  N=10
  distinct raw texts: 10
  distinct signatures: 1
   10 x task_passed=True action=approve_4.2
  pass@1: 10/10
  pass@10 ceiling (unbiased, N=10): 1.000

These are this program’s own figures from its seeded simulation. The chapter reports no measured run here; its 12/20 and 9/10 figures are constructed.

The projection decides what you measure

Twenty raw texts would give twenty groups, no shape at all. The signature projects each output onto the properties the debugging question needs, so two outputs differing in wording but citing the same evidence land in the same group. The representation you choose determines the distribution you think you measure, so the projection is fixed first:

@dataclass(frozen=True)
class Signature:
    schema_valid: bool
    supported_claims: int
    unsupported_claims: int
    action: str | None
    task_passed: bool

Two modes in a twenty-draw series is the bimodal shape the chapter calls H2, and the probe collapses to one mode, which is what H1 predicts when the cause is sampling spread. One mode is not proof: the same collapse appears for a serving-plumbing change or a silent revision drift, which the chapter separates with arms this program does not run.

Mechanism and limitations

Documented: the chapter states that a single draw from a mixed distribution carries almost no information about the distribution, that the debugging object is the answer-distribution rather than the answer, that the behavioural signature must be fixed before the trials or the histogram is a Rorschach test, that pass@k is a selection ceiling and never describes shipped behaviour, and that pass@1 carries raw counts.

Observed: the run above. The twenty baseline draws produced twenty distinct raw texts and two signatures, and the ten probe draws produced ten raw texts and one signature. pass@1 was 14/20 on the baseline and 10/10 on the probe, and pass@10 was 1.000 on both, the ceiling not the rate.

Not shown. No model is called. The draws come from a seeded simulation with a fixed 0.6 probability, so the numbers are properties of that simulation, not of any model, endpoint or prompt. The H3 and H4 arms are not implemented, so a one-mode probe fits spread and drift alike.

Read the chapter .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.