The nearest neighbour can be wrong, and a readout can be right

The nearest neighbour is a guess made with a metric that was never asked what the relation was. The chapter’s claim is that a supervised readout over the same vectors finds the relation the metric is blind to, and that a benchmark which mines its negatives adversarially is the honest way to see the gap.

Run it

git clone https://github.com/ernanhughes/relate.git
cd relate
git checkout 1de8c95b3995791042d6eb1c150c64c7cdb3492f
python -m venv venv
venv/Scripts/pip install -e ".[dev]"
venv/Scripts/python benchmarks/hard-negatives/run.py --check

--check verifies the frozen artifacts instead of overwriting them. Python 3.10 or later. No model download, no network.

Expected outcome

MATCH

The frozen artifacts carry both halves of the comparison. The relation readout:

{
  "metrics": {
    "accuracy": 0.725,
    "loss_rate": 0.275,
    "mean_margin": 0.18869503123199047,
    "tie_rate": 0.0,
    "total": 200
  },
  "scorer": "relation_chebyshev:complexity,depth,call_sites",
  "task": "hard_negative_ordering"
}

And the margin the raw geometry does not have:

{
  "accuracy_delta": 0.625,
  "mean_margin_delta": 0.42200305074086725,
  "tie_rate_delta": 0.0
}

An accuracy delta of 0.625 on a task where raw cosine scores 0.100 is the whole claim in one number.

The historic result, and why it is not recomputed

The repository also preserves the original CodeBERT result, and it is careful about what it is:

{
  "status": "EXTERNAL",
  "note": "Preserved prose evidence, NOT recomputed here. The frozen CodeBERT embeddings and AST coordinates behind these figures are absent from this machine (no similarity_is_relative checkout, no .writer replay assets), so digit-level recomputation is blocked. The synthetic mirror in expected/report-*.json reproduces the observation structure through the generic evaluator instead.",
  "results": {
    "chebyshev_relation_space": 0.7328515625,
    "raw_cosine": 0.532458984375,
    "raw_euclidean": 0.533314453125
  }
}

The note is the part that matters. The historic figures are 0.733 for the relation readout against 0.532 and 0.533 for raw geometry, and they are marked EXTERNAL because the assets needed to recompute them are absent. The mirror reproduces the observation structure and the evidence map says so in words:

Mirror: relation 0.725 vs cosine/euclidean 0.100 through one evaluator. Historic: 0.733 / 0.532 / 0.533, not recomputed (CodeBERT assets absent).

The mirror’s raw geometry scores much lower than the historic one, and the repository says why: the mirror’s negatives are adversarially mined, which is the regime the book targets.

Mechanism and limitations

Documented: the case adapter, the seeded synthetic vectors and the frozen artifacts are in benchmarks/hard-negatives/; the historic reference and the mirror’s relationship to it are in benchmarks/hard-negatives/expected/historic-reference.json; the chapter mapping and the limitations are in evidence/BOOK-EVIDENCE-MAP.md.

Observed: the run above. The artifacts reproduce byte-for-byte. The relation readout scored 0.725 accuracy on 200 adversarial orderings with no ties, and the delta over raw geometry was 0.625.

Not shown. The mirror’s vectors are synthetic, and the historic CodeBERT figures are preserved prose evidence that this machine cannot recompute. The mirror’s negatives are adversarially mined, so the raw-geometry figures are harsher than the fixed historic pairs, and the two numbers are not interchangeable. A supervised readout needs labels: this run says nothing about what happens when the relation is not known in advance, and the relation here is a three-coordinate Chebyshev distance over AST features, not a semantic relation between sentences.

Canonical source and the evidence map .