Part IV — Measuring the Representation

Two pairs, same cosine, different situations

Pair 1:  cos = 0.78
  the query's nearest neighbor scores 0.78; the 2nd scores 0.44
  the region is sparse; the ranking is stable under paraphrase
  → a confident, isolated match

Pair 2:  cos = 0.78
  the nearest neighbor scores 0.78; the 2nd, 3rd, 4th score 0.77, 0.76, 0.76
  the region is a dense hub; re-embedding the query with a typo flips the top result
  → a coin toss dressed as a match

The scalar is identical. The situations are not. Everything that distinguishes them is geometric information the cosine threw away.

Is the thing we care about — “are these a match?” — actually one number, or does it take several to describe it honestly?

And a sharper version, because some of the signals below come from a second model rather than from the geometry:

How much of its own uncertainty can an embedding space diagnose before another model is introduced?

Signals available from the same pair

Two kinds. Geometric signals are computed from the embedding space alone — nothing but the vectors and the index. External signals bring in a second model or an oracle. The distinction matters: the claim “geometry carries more than one scalar” must rest on the geometric signals; the external ones are a separate, later escalation.

Geometric (from the embedding space alone):

• Similarity score. The baseline. cos(q, d).
• Margin. cos(q, d₁) − cos(q, d₂): how far the top result beats the runner-up. Small margin = ambiguous, regardless of absolute score.
• Local density around the query. Distance to the query’s kth neighbor. In a hub region, high absolute similarity is cheap.
• Hubness / in-degree. How many other items’ neighbor lists the retrieved item sits in (Chapter 6).
• Rank stability under perturbation. Re-embed the query with small changes (typo, paraphrase, truncation). Does the top result stay? Fraction that survives = stability.
• Participation ratio of the top-k similarities. Is the retrieved set dominated by one item or spread across many near-ties?
• Neighborhood overlap across k. Does the top-5 stay a subset of the top-20 as k grows, or does the set churn?
• Sensitivity. If you change a meaningful part of the query (a date, an entity), does the retrieved set change? If not, the match may be topical only (the Hallucination book’s “safe but useless,” in retrieval form).
• Query difficulty. Query norm, query length, whether the query embeds into a dense or sparse region.

External (a second model or an oracle):

• Cross-encoder agreement. Does a cross-encoder reranker agree with the bi-encoder ranking?
• Second-encoder agreement. Does a different embedding model put the same item on top (Chapter 16)?
• Verification result. Does a claim-checking step confirm the retrieved passage supports the query (the Hallucination book’s territory).

The geometric signals are not redundant with each other. Margin and absolute score decorrelate on hard cases. Stability and density capture different pathologies. On the RELATE hard-negative subset (Wave 1 row 1.12), a small model of geometric-only signals lifts balanced accuracy from 0.76 (score alone) to 0.90; an NLI cross-encoder added on top changes nothing (−0.003).

The diagnostic vector

Instead of

match_confidence = 0.78

produce

R(q, d) = {
  # geometric — from the embedding space alone
  score:       0.78
  margin:      0.02      LOW
  density:     hub        (query in top-5% densest region)
  stability:   0.4        (top result flips on 60% of perturbations)
  sensitivity: entity-swap does not change top-3   FLAT
  # external — a second model was consulted
  alignment:   bi/cross disagree
}

The geometric block is available for free on every retrieval. The external block costs a second model call, so a policy consults it only when the geometric block is already ambiguous (low margin, hub, unstable).

Now a policy (Chapter 12, and the Hallucination book’s approach) can route:

    flowchart TD
    G["geometric block — free on every retrieval"] --> D{"read the signals"}
    D -->|"score high, margin high, stable"| A[accept]
    D -->|"score high, margin low"| R["rerank / escalate to the external block"]
    D -->|"stability low"| W["widen the query, re-retrieve"]
    D -->|"sensitivity flat"| V["the match is topical — verify the claim"]
  

What collapses and what does not

Some applications genuinely only need the scalar: coarse deduplication, “show me more like this,” recommendation where errors are cheap. For those, one number and a calibrated threshold (Chapter 14) is right, and a diagnostic vector is over-engineering.

The vector earns its cost when errors are expensive and the easy/hard gap is large — RAG feeding a model that will state the retrieved content as fact, record linkage on people, safety filters. There, the extra signals are the difference between “accept at 0.78” and “accept at 0.78 with margin 0.02 in a hub, so don’t.”

Demonstration: does the vector beat the scalar on hard negatives?

MEASURED on RELATE v0.1, Wave 1 row 1.12 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/signal-ablation.json. 1,206 (grade-3 positive vs hard-negative) examples; 5-fold balanced accuracy; model all-mpnet-base-v2, external model cross-encoder/nli-deberta-v3-base.

Task: separate the grade-3 correct answer from the query’s hard negatives.

predictor                                                  balanced accuracy
GEOMETRIC ONLY (no second model)
  score only                                                     0.76
  + margin + local density + top-k spread + in-degree + rank     0.90
+ EXTERNAL (a second model is now consulted)
  + NLI cross-encoder entailment probability                     0.897   (−0.003)

MEASURED: the six geometric signals lift balanced accuracy from 0.76 to 0.90 — a +0.14 gain, entirely on the cases the raw score gets wrong. Adding an NLI cross-encoder on top adds nothing (−0.003, within noise). On RELATE the geometry diagnoses its own hard cases, and the obvious “second model” — an off-the-shelf NLI cross-encoder — is not the right escalation for these distinctions (it is confused by the same question/negation mismatch seen in Chapter 10). The phenomenon “is this the right match?” is better described by five or six numbers than by one; a task-matched verifier would be the next step, not a generic one.

What this chapter establishes and what it does not

Establishes: a similarity scalar discards geometric information (margin, density, hubness, stability, sensitivity) that is decision-relevant on hard cases, and that the geometry can diagnose much of its own uncertainty before any second model is called; external signals (cross-encoder, second encoder, verification) are a further, costlier escalation, not part of the “geometry carries more than one scalar” claim; a diagnostic vector supports routing that a threshold cannot; the vector is worth its cost when errors are expensive and the easy/hard gap is large, and is over-engineering otherwise.

Does not establish: a fixed universal signal set (it is application-specific), or that the vector always beats the scalar (on easy cases it does not, meaningfully). It establishes that “match confidence = one number” is a modeling choice to make consciously.

Lab 15: scalar vs. vector on your hard cases

PROPOSED, not executed.

Setup. 200+ hard (query, correct, distractor) triples for your task.

Task.

1. Compute the geometric signals — score, margin, local density, hubness, perturbation stability (5 perturbations), sensitivity — per triple.
2. Fit a simple classifier (logistic regression / small tree) to predict “correct” from subsets.
3. Report accuracy for: score only; score+margin; all geometric (the headline — how far does geometry alone get?); then all geometric + one external signal (cross-encoder agreement) to measure the marginal lift of introducing a second model.
4. Inspect 10 cases where the vector is right and the scalar is wrong.

Success criterion. Two numbers: how much accuracy the geometry recovers on its own, and the marginal lift from the first external model. Then a decision: does your application need the vector, and does it need the external tier or only the geometric one?

Companion component: the signal bundle

signal_bundle(q, results):
  geometric:                          # embedding space alone — always available
    score:        top-1 similarity
    margin:       top1 - top2
    density:      query distance-to-kth-neighbor (percentile)
    hubness:      in-degree of the top result
    stability:    fraction of perturbations preserving top-1
    sensitivity:  does a meaningful query edit change top-3?
  external:                           # a second model was called — populated on escalation only
    cross_encoder_agreement: agreement(bi_rank, cross_rank) | null
    second_encoder_agreement: agreement(model_A_top1, model_B_top1) | null
    verification: {supported | contradicted | unverified} | null
  verdict:        {accept | rerank | rewiden | verify}   from a policy over the above

The Observatory computes the geometric block on every retrieval. It populates the external block only when the geometric block is ambiguous (low margin, hub, unstable) and a policy asks for the escalation — so the cost of a second model is paid only where geometry could not decide.

Failure modes

• One number for an expensive decision. Margin 0.02 in a hub looks identical to margin 0.3 in open space if you only log the score.
• A diagnostic vector for a cheap decision. Over-engineering; a calibrated threshold is enough.
• Signals that are actually redundant. Check decorrelation on your hard cases before adding a signal.
• Perturbation stability with meaningless perturbations. Random character noise tests robustness to noise, not to paraphrase; choose perturbations that match real query variation.

What this chapter established

• A similarity scalar discards geometric signal — margin, density, hubness, stability, sensitivity — all decision-relevant on hard cases.
• Geometric signals (embedding space alone) are separated from external signals (a second model or an oracle); the chapter’s claim rests on the geometric ones, and the sharper question is how much uncertainty geometry diagnoses about itself before a second model is called.
• The diagnostic vector R(q, d) and the routing it enables.
• The vector pays off when errors are expensive and the easy/hard gap is wide; otherwise the scalar plus a calibrated threshold is correct.
• On RELATE hard negatives (Wave 1 row 1.12, measured), geometric-only signals lift correct-match balanced accuracy from 0.76 to 0.90; a generic NLI cross-encoder on top adds nothing — the external increment depends on the verifier being matched to the missing distinction.
• The signal bundle: geometric block computed on every retrieval, external block populated only on escalation.

Next

Part IV measured one representation from every angle. Part V introduces a second model — and finds that the same text, embedded by a different encoder, does not just get different numbers; it enters a different universe with its own coordinate system.