Example

A threshold calibrated on one domain does not transfer to another

Run the calibration benchmark and see the equal-error rate move from 0.159 on ordinary pairs to 0.296 on hard ones, and the staleness check name what changed.

A threshold is a number, and the chapter’s claim is that a number is only meaningful next to the distribution it was chosen on. The benchmark here calibrates on two domains and then reports what a staleness check notices when the inputs move.

Run it

git clone https://github.com/ernanhughes/relate.git
cd relate
git checkout 1de8c95b3995791042d6eb1c150c64c7cdb3492f
python -m venv venv
venv/Scripts/pip install -e ".[dev]"
venv/Scripts/python benchmarks/calibration/run.py --check

--check verifies the frozen artifacts instead of overwriting them. Python 3.10 or later, NumPy only, no model download, no network.

Expected outcome

MATCH

The summary is the whole result, and it is two domains side by side:

{
  "domain_eer_shift": 0.10881996292004725,
  "hard": {
    "ambiguity_fraction": 0.47833333333333333,
    "auc": 0.7742631111111111,
    "eer_rate": 0.296,
    "eer_threshold": 0.5336699988016802
  },
  "ordinary": {
    "ambiguity_fraction": 0.12633333333333333,
    "auc": 0.9249417777777778,
    "eer_rate": 0.15866666666666668
  },
  "seam_auc": 1.0,
  "staleness_changed": [
    "space_hash",
    "domain",
    "scorer"
  ]
}

Three things to read. The AUC drops from 0.925 to 0.774 when the pairs get hard. The equal-error rate moves by 0.109, which is a threshold that does not transfer. And the staleness check names what moved, rather than saying “the results are stale”: the space hash, the domain and the scorer.

A threshold is not a property of a model

The seam_auc of 1.0 is the control: the check itself can detect a seam when there is one, which is what makes staleness_changed worth reading. The chapter’s practical claim follows: a calibration is a claim about a distribution, and the distribution is identified by the space, the domain and the scorer. Change any of the three and the number needs recomputing, which is what staleness_changed reports.

The ambiguity_fraction is the other half. Nearly half the hard pairs are ambiguous, against an eighth of the ordinary ones, so part of the AUC drop is the task being ill-posed rather than the scorer being wrong. A calibration report that gave you only the AUC would hide that.

Mechanism and limitations

Documented: the two domains, the operating points, the ROC artifacts and the staleness check are in benchmarks/calibration/; the chapter mapping is in evidence/BOOK-EVIDENCE-MAP.md.

Observed: the run above. The artifacts reproduce byte-for-byte. The equal-error rate moved from 0.159 to 0.296 between the ordinary and the hard domain, the AUC from 0.925 to 0.774, and the staleness check named the space hash, the domain and the scorer as the changed inputs.

Not shown. The pairs are synthetic and seeded, so this is a mirror of the calibration question, not a measurement of a real encoder’s scores. The equal-error rate is one operating point summary; it says nothing about which threshold is right for a given cost, and the AUC says nothing about whether the scores are well ordered in the region that matters. The staleness check reports that inputs changed; it does not report how much the calibration should have moved, and it cannot detect a change that leaves all three identifiers intact.

Canonical source and the evidence map .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.