Part II — Inside the Space

Same corpus, three shapes

Embed the same 1,173 RELATE items with five models. Do not look at task scores yet. Look at the shape (measured — Wave 1 row 1.4, experiments/embeddings-from-first-principles/wave1/artifacts/anisotropy.json):

                          MiniLM-L6   mpnet-base   mxbai-large   bge-large   bge-small
dimension                     384         768         1024          1024        384
mean random-pair cosine       0.06        0.08         0.34          0.40        0.45
top singular-value share      0.07        0.08         0.09          0.09        0.10
effective rank                 259         387          425           434         271
distance concentration*       0.08        0.09         0.09          0.11        0.10

*ratio of standard deviation to mean of pairwise distances — smaller means “everything is roughly equidistant.”

The anisotropy varies 7× — MiniLM and mpnet spread vectors nearly isotropically (a random pair has cosine ~0.07), while the BGE and mxbai models pack the corpus into a cone where every random pair scores 0.3–0.5. These differences exist before any question of which model is “better.” What did not vary much on RELATE: no model is dominated by a single principal component (top singular-value share is ~0.08 everywhere), and all five are distance-concentrated (relative spread ~0.10) — so the “one giant nuisance direction” pathology is not what separates these particular models; the similarity origin is.

What geometry does a model actually impose on a corpus, and how do you describe it in a way that transfers across models?

The shape descriptors

• Anisotropy. Deviation from uniform directional spread. Measured by mean random-pair cosine, or by the ratio of the largest to the mean eigenvalue of the covariance. High anisotropy ⇒ a non-zero similarity origin and compressed dynamic range.
• Distance concentration. In high dimensions, ‖x − y‖ for random x, y concentrates around a single value; the relative contrast (max − min)/min shrinks. If contrast is tiny, nearest-neighbor rankings are decided by noise.
• Density and its variation. From Chapter 6, but now as a global descriptor: the coefficient of variation of local density.
• Dominant directions. The top few principal components. Often these encode nuisance: sentence length, punctuation, language, formatting. Removing them (mean-centering, “all-but-the-top,” whitening) frequently improves semantic similarity.
• Covariance structure. Is the covariance close to a scaled identity (isotropic) or highly structured? Whitening transforms it toward identity.
• Cluster geometry. Number, size, separation, and shape of dense regions.

Whitening and isotropy: the standard repair

Post-processing pipelines that improve raw embedding similarity mostly do the same thing:

    flowchart TD
    RAW["raw embedding space"] --> C["1. subtract the mean vector — kills the dominant offset direction"]
    C --> P["2. remove the top 1-k principal components — kills nuisance directions (length, punctuation, formatting)"]
    P --> W["3. rescale remaining directions to equal variance — whitening / BERT-flow / BERT-whitening"]
    W --> D{"did semantic ranking improve?"}
    D -->|"large gain"| N["raw geometry carried heavy nuisance structure (non-contrastive encoder, lopsided spectrum)"]
    D -->|"neutral or worse"| CL["raw geometry was already clean (contrastive encoder) — rescaling only amplified low-variance noise"]
  

Effect: mean random-pair cosine drops toward 0, distance contrast rises, semantic ranking improves — sometimes by a lot on models that were not trained with a contrastive objective. Models that were trained contrastively are often already near-isotropic and gain little.

This is why “does whitening help?” is a diagnostic: if it helps a lot, your model’s raw geometry was carrying heavy nuisance structure; if it hurts (as it did for all five encoders measured in row 2.9), the raw geometry was already clean and the rescaling only amplified low-variance noise.

Shape is not quality — but it predicts failure modes

• High anisotropy ⇒ thresholds are fragile (Chapter 14), raw scores compressed, cross-model comparison harder (Chapter 16).
• Severe distance concentration ⇒ kNN unstable, approximate search less accurate, hubness worse (Chapter 6).
• One giant principal component ⇒ that component may be nuisance; check what sorting by it reveals.
• Very low effective rank + low ID ⇒ possible mode collapse; check whether distinct inputs get distinct vectors.

A model can have an ugly shape and good task scores (the objective found signal despite the nuisance), or a clean shape and mediocre scores. Shape tells you where the risks are, not whether it works.

Demonstration: whitening the RELATE space

MEASURED on RELATE v0.1, Wave 2 row 2.9 — artifact experiments/embeddings-from-first-principles/wave2/artifacts/whitening-gain.json.

The five locally-available encoders are all contrastively trained and L2-normalize their output. Running the standard post-processing pipeline on all-mpnet-base-v2:

stage                              nDCG@10   hard-neg margin   mean random-pair cosine
raw                                  0.952        0.113               0.08
mean-centered                        0.952        0.113               0.07
drop top 3 PCs                        0.948        0.111               —
BERT-whitening (keep 95%-var, ÷σ)     0.910        0.107               ~0
full whitening (all 768 axes ÷σ)      0.527          —                 ~0

MEASURED: for a model whose raw geometry is already near-isotropic, whitening does not help — it hurts. BERT-whitening costs ~4 points of nDCG@10; naive full whitening is catastrophic (it divides the ~380 near-zero-variance tail axes by ≈0 and lets noise dominate). Centering alone is neutral. The diagnostic still works, with the sign flipped: a non-positive whitening gain says this model’s raw geometry carries no nuisance bulk to remove. Where whitening does recover ~10 points (the BERT-flow / BERT-whitening results in the literature) the input is a non-contrastive encoder with a genuinely lopsided spectrum — none of the locally-cached models is that. Hard-negative resolution barely moves either way: shape repair never adds a distinction the model did not encode (the Chapter 4 lesson again).

What this chapter establishes and what it does not

Establishes: a transferable vocabulary for describing embedding-space shape (anisotropy, concentration, dominant directions, covariance, cluster geometry); that whitening/centering is a nuisance-removal operation whose benefit diagnoses the raw geometry; that shape predicts failure modes but not task quality.

Does not establish: that any shape is “correct,” or that whitening always helps (it can hurt contrastively-trained models). It establishes that you should describe a model by the geometry it produces, not its dimension.

Lab 8: shape comparison across models

PROPOSED, not executed.

Setup. One corpus (1,000+ items). Three or more embedding models. Same items, same code.

Task.

1. For each model compute: mean random-pair cosine, top-1 singular-value share, effective rank, distance concentration (std/mean of pairwise distance), density CV.
2. Apply mean-centering → drop-top-k → whitening. Re-measure shape and retrieval at each stage.
3. Plot each model’s singular-value spectrum on one axis.

Success criterion. A paragraph per model: describe its geometry in shape terms, predict its main failure mode, and say whether whitening helped (and what that implies about its training).

Companion component: the shape profile

shape_profile(space, corpus):
  anisotropy:            mean_random_pair_cosine, eigenvalue_ratio
  concentration:         std_over_mean_pairwise_distance
  dominant_directions:   top-k PCs + a human note on what each sorts by
  whitening_gain:        Δ Recall@10 from centering + whitening
  predicted_risks:       [threshold_fragility | knn_instability | mode_collapse | ...]

The Observatory computes a shape profile per (model, corpus) and uses whitening_gain to decide whether to store raw or whitened vectors.

Failure modes

• Comparing models by dimension. d is not shape. Two 768-dim models can be geometrically unalike.
• Whitening a contrastive model reflexively. Measure the gain; it may be negative.
• Ignoring the dominant PC. Sort by it once. If it is “sentence length,” you have a nuisance axis inflating your scores.
• Reading a clean shape as good quality. Shape ≠ task performance; still run Chapter 13’s evaluation.

What this chapter established

• Shape descriptors: anisotropy, distance concentration, density variation, dominant directions, covariance structure, cluster geometry — a vocabulary that transfers across models.
• The same corpus takes visibly different shapes under different models, independent of task quality.
• Whitening/centering removes nuisance structure; the signed size of its effect diagnoses the raw geometry (positive → nuisance was present; non-positive, as measured for all five encoders in row 2.9 → geometry already clean); it never adds semantic resolution.
• The shape profile: per (model, corpus), with predicted risks and a raw-vs-whitened storage decision.

Next

Part II characterized the space at rest. Part III puts it to work. The next chapter implements retrieval from first principles — embed, score, sort — with no vector database, so the primitive operation and its parameters are completely visible before any index is introduced.