Part II — Inside the Space

What does dimension 173 mean?

Pull the 173rd coordinate of every vector in your corpus and sort. You get a list of documents ordered by… something. Occasionally a coordinate is weakly interpretable — “this one seems higher for questions” — but usually the sorted list has no human-nameable theme. Dimension 173 does not mean “formality” or “sentiment” or “is about sports.”

And yet the vector works. Similarity search returns sensible results. The structure is there; it is just not living in the individual axes.

If no single coordinate carries a nameable meaning, where is the meaning, and what does that imply about how we should treat the numbers?

Distributed representation

A concept is encoded as a pattern across many coordinates, and each coordinate participates in many concepts. This is a distributed representation, and it is a design property, not an accident:

• It is efficient: d coordinates can express far more than d concepts.
• It degrades gracefully: corrupting a few coordinates blurs everything slightly rather than deleting one thing.
• It makes single-axis interpretation a category error: asking “what does dimension 173 mean” is like asking “what does neuron 4,000,112 in your visual cortex mean.”

Basis dependence and rotation invariance

Here is the experiment that reframes the whole chapter.

Take your embedding matrix X (n items × d dims). Generate a random orthogonal matrix R (d × d). Compute X' = X R.

Now check:

all pairwise cosine similarities:   IDENTICAL
all pairwise euclidean distances:   IDENTICAL
nearest-neighbor lists:             IDENTICAL
retrieval results:                  IDENTICAL

every individual coordinate:        COMPLETELY DIFFERENT
"dimension 173":                    now a mix of all old dimensions

Rotation preserves every inner product and distance, so it preserves everything retrieval and clustering depend on — while scrambling the coordinates entirely. The coordinate system is arbitrary up to rotation (and often reflection and permutation).

Coordinates are not semantics. The relationships between coordinates — inner products, distances, angles — carry the usable structure.

Anything you claim about “the meaning of a dimension” must survive an arbitrary rotation of the space. Almost no such claim does.

Superposition and why axes are messy

Models under capacity pressure pack more features than they have dimensions by storing them in non-orthogonal directions — accepting a little interference between rarely-co-occurring features in exchange for representing more of them. This is superposition. Consequences:

• Feature directions are not axis-aligned and not mutually orthogonal.
• Some directions are “polysemantic”: they light up for several unrelated concepts.
• Sparse-coding methods (dictionary learning, sparse autoencoders) can sometimes recover more interpretable directions than the raw axes — evidence that the interpretable structure is real but rotated away from the basis.

Anisotropy: the space is not evenly used

In many trained embedding spaces, vectors are not spread evenly over all directions. They occupy a narrow cone: there is a dominant direction (often close to the mean vector) that almost every vector has a large positive component along.

mean cosine between two RANDOM items:   ~0.0   in an isotropic space
                                        0.3–0.6 in a typical raw contextual space

If two unrelated sentences already have cosine 0.4 just from the shared cone, then a “relevant” cosine of 0.6 is a much weaker signal than it looks. Fixes — mean-centering, whitening, standardizing the spectrum — are covered in Chapter 8; the point here is that the raw coordinates sit inside a lopsided distribution.

Demonstration: rotate RELATE, nothing changes; look at an axis, nothing is there

MEASURED on RELATE v0.1 — the anisotropy cone from Wave 1 row 1.4 (wave1/artifacts/anisotropy.json), the rotation check from Wave 2 row 2.1 (wave2/artifacts/rotation-invariance.json).

Embed all 1,173 RELATE items and compute the mean cosine over 5,000 random item pairs — the “similarity origin” of the space:

model                    mean random-pair cosine   % of random pairs with cosine > 0   effective rank / dim
all-MiniLM-L6-v2                  0.06                          66%                        259 / 384
all-mpnet-base-v2                 0.08                          73%                        387 / 768
mxbai-embed-large-v1             0.34                         100%                        425 / 1024
bge-large-en-v1.5                0.40                         100%                        434 / 1024
bge-small-en-v1.5               0.45                         100%                        271 / 384

For the three anisotropic models every random pair has positive cosine and the origin sits at 0.34–0.45 — a “relevant” cosine of 0.6 is then barely above the floor. MiniLM and mpnet are close to isotropic (origin ~0.07). The effective rank (entropy of the singular-value spectrum) is a few hundred regardless of the nominal dimension.

Now the rotation check. Apply three independent random rotations to every vector and re-run retrieval: nDCG@10 changes by 4·10⁻⁷ — rotation-invariant to numerical precision, exactly as the algebra says. Then look at the single highest-variance coordinate: before rotation it correlates weakly with sentence length (r ≈ 0.29); after rotation a different coordinate is on top and the length signal has scattered across the basis. The geometry is unchanged; the “meaning of axis 7” is an artifact of the basis.

MEASURED: retrieval is exactly rotation-invariant; single-axis interpretation is rotation-fragile; the raw cosine scale has a non-zero, model-specific origin (0.07 to 0.45). Interpreting an absolute similarity value without subtracting that origin, or naming a coordinate, is reading a number the space does not actually contain.

What this chapter establishes and what it does not

Establishes: representations are distributed; the coordinate basis is arbitrary up to rotation; retrieval-relevant quantities (inner products, distances, neighborhoods) are exactly rotation-invariant; superposition explains messy axes; anisotropy means the raw similarity scale has a non-zero origin.

Does not establish: that dimensions are never interpretable (some models and some training regimes yield partly interpretable axes), or that anisotropy is always harmful (Chapter 8 measures it). It establishes that interpretability claims must survive rotation, and most do not.

Lab 5: the rotation test

PROPOSED, not executed.

Setup. Embed 1,000 items. Pick your retrieval metric and record a baseline (Recall@k, MRR).

Task.

1. Generate 3 random orthogonal R (e.g. QR of a Gaussian matrix). Re-score. Confirm invariance.
2. For each of the 5 highest-variance coordinates, list the 10 items scoring highest and lowest. Try to name the axis.
3. Repeat step 2 after one rotation. Try again.
4. Measure mean pairwise cosine on 5,000 random pairs (the anisotropy baseline).

Success criterion. State the rotation-invariant quantities and the rotation-fragile ones, and give this space’s “similarity origin” (mean random-pair cosine) — the number you must subtract before a cosine means anything.

Companion component: the basis-invariance guard

The Observatory tags every derived quantity as invariant or fragile:

quantity_class:
  rotation_invariant:  [cosine, dot, l2, knn, clusters, retrieval_metrics]
  rotation_fragile:    [per-axis stats, "dimension X means Y", axis-aligned pruning]
anisotropy:
  mean_random_pair_cosine:  <float>     # the similarity origin
  dominant_direction_share: <variance fraction of top singular value>

Any report that makes a rotation-fragile claim is flagged for review.

Failure modes

• “Dimension 42 is the sentiment dimension.” Unless it survives a random rotation, it is an artifact of the basis.
• Axis-aligned dimensionality pruning by variance. Dropping low-variance coordinates is basis-dependent; drop low-variance principal components instead (Chapter 7).
• Reading raw cosine on an anisotropic space. 0.6 might be barely above the 0.5 floor.
• Trusting a single sparse-autoencoder feature as ground truth. It is a hypothesis about a direction, not a labeled fact.

What this chapter established

• Individual coordinates rarely carry nameable meaning; representations are distributed.
• The rotation test: cosine, distance, neighborhoods, and retrieval metrics are exactly invariant to an orthogonal change of basis; per-axis interpretations are not.
• Superposition: features are stored in non-orthogonal, sometimes polysemantic directions, off the axes.
• Anisotropy: raw embedding spaces often occupy a cone, so the similarity scale has a non-zero origin you must measure.
• The basis-invariance guard: quantities tagged invariant vs. fragile; fragile claims flagged.

Next

If the meaning lives in relationships and neighborhoods rather than axes, then we should look directly at neighborhoods. The next chapter examines local structure — nearest neighbors, density, hubs, boundaries — and finds it is stranger and less uniform than the global picture suggests.