Turn a delta into an operator, and measure what it costs

A delta between two embeddings is not a transformation. The chapter’s claim is that to act on a difference you need an operator, and that choosing one is a decision with a measurable cost. The benchmark here fits four candidate operators per transformation class and records which one was selected and what each scored.

Run it

git clone https://github.com/ernanhughes/relate.git
cd relate
git checkout 1de8c95b3995791042d6eb1c150c64c7cdb3492f
python -m venv venv
venv/Scripts/pip install -e ".[dev]"
venv/Scripts/python benchmarks/operators/run.py --check

--check verifies the frozen artifacts instead of overwriting them. Python 3.10 or later, NumPy only, no model download, no network.

Expected outcome

MATCH

The summary records, per transformation class, the four fitted operators and the one that was selected:

{
  "active_to_passive": {
    "fidelities": {
      "affine": 0.9374245432071066,
      "constant_delta": 0.9947041432568534,
      "identity_map": 0.9948496244895532,
      "linear": 0.944746871949898
    },
    "n_eval": 15,
    "n_train": 30,
    "native_cosine": 0.9948496244895532,
    "outcome": "selected",
    "selection": "active_to_passive/identity_map",
    "train_linear_diagnostic": 1.0
  }
}

The interesting part is the class, not the winner. For active_to_passive the identity map wins, because the native cosine is already 0.995 and there is nothing to correct. For claim_weakened the ordering changes:

  "claim_weakened": {
    "fidelities": {
      "affine": 0.7968530172311908,
      "constant_delta": 0.9679428388268928,
      "identity_map": 0.8313307541081256,
      "linear": 0.3065985524844288
    }

Here a constant delta (0.968) beats the identity map (0.831), and the linear operator collapses to 0.307. One number would have hidden that; four numbers per class show which kind of transformation the delta actually is.

The selection is recorded, not implied

Every class carries an outcome, a selection string, the number of training and evaluation cases, and the hash of each case set. A reader can see which operator was chosen, on how much data, and can tell that the two case sets are different ones. The train_linear_diagnostic of 1.0 is there so a reader can tell the linear fit was not simply broken on the training set.

Mechanism and limitations

Documented: the nine transformation classes, the 260 real transformation pairs and the four operator families are in benchmarks/operators/; the selection rule and the per-class limitations are in evidence/BOOK-EVIDENCE-MAP.md.

Observed: the run above. The artifacts reproduce byte-for-byte. For active_to_passive the identity map was selected at a fidelity of 0.995; for claim_weakened the constant delta was selected at 0.968 while the linear operator reached only 0.307.

Not shown. The operator families are four hand-chosen forms, and a transformation outside them is outside the benchmark. The evaluation sets are small: 5 to 15 cases for the classes shown, and a fidelity measured on 5 cases is a coarse number. A fidelity of 0.995 against the native cosine says the operator reproduces the observed delta on those cases; it says nothing about whether applying the operator to a new delta would be right, and the benchmark does not test that.

Canonical source and the evidence map .