Part II — Inside the Space

A 1,536-number vector that isn’t

Your embedding model outputs 1,536 floats. Take 50,000 of its vectors, stack them, run SVD, and plot the singular values. Typically:

first ~20 singular values:     large, falling fast
next ~150:                     moderate, gentle slope
remaining ~1,350:              tiny, near the noise floor

Reconstruct every vector from just the top 200 components. Re-run retrieval. Recall@10 barely moves. The other ~1,300 dimensions were carrying almost no variance that mattered for the task.

The advertised dimension is 1,536. The number of directions the representation actually uses, on this corpus, is a few hundred.

How large is the space a representation actually occupies — and which of the many “dimension” numbers should you report?

Nominal vs. effective — six measures

• Nominal dimension d. The length of the array. A model spec, not a property of the data.
• Rank. The number of non-zero singular values. Almost always = d for real data (noise fills every direction), so rank alone is uninformative.
• Effective rank (spectral entropy). exp(H(p)) where p are the normalized singular values and H is Shannon entropy. Answers “how many directions, weighted by how much variance they carry?” Typical contextual spaces: tens to low hundreds.
• Participation ratio. (Σσᵢ²)² / Σσᵢ⁴. Another “how many directions matter” scalar, more sensitive to the tail. Often close to effective rank.
• Intrinsic dimension (ID). The dimension of the manifold the data lies on, estimated locally (e.g. by the ratio of distances to the 1st and 2nd neighbors — the TwoNN / MLE estimators). Usually much smaller than effective rank — often 5–30 for sentence embeddings.
• Entropy / stable rank. ‖X‖_F² / ‖X‖_2² — variance spread relative to the largest direction. A quick anisotropy-aware capacity proxy.

These measure different things. Effective rank is about the global variance spectrum. Intrinsic dimension is about the local geometry of the data cloud. A space can have effective rank 150 and intrinsic dimension 12.

Why the gap exists

• Correlated features. Many learned directions co-vary; PCA collapses them.
• Anisotropy (Chapter 5). A dominant direction eats a large share of variance, leaving less for the rest.
• The manifold is thin. Real text does not fill the space; it traces a low-dimensional surface embedded in it.
• Training does not penalize unused dimensions. There is no pressure to spread information across all d axes.

Why it matters practically

• Large redundancy is typical; the safe amount of compression is task-specific. Modern embedding spaces carry far more nominal dimensions than any one task needs — keeping the first quarter of the dimensions is often near-lossless averaged over tasks (Tsukagoshi & Sasano, 2025). But “averaged over tasks” hides an order-of-magnitude spread: on the same embeddings, classification and clustering can survive reduction to under 1% of the dimensions while retrieval and semantic-similarity degrade at 10–25%. Redundancy tells you some compression is safe. Only a per-task retention curve (below) tells you how much.
• Effective rank is a model comparison axis. Two models at d = 768 can have effective ranks of 90 and 300. The second is using its capacity; the first may be wasting it or may be more efficient — depends on task performance (Chapter 8, 13).
• Low intrinsic dimension warns about redundancy and mode collapse. If ID drops to 3, the model may be mapping everything to a few templates (related to the “safe but useless” failure in the Hallucination book). The intrinsic dimension of a trained representation is tied to how well it does its job — in vision networks the last layer’s ID predicts test accuracy (Ansuini et al., 2019).
• Johnson–Lindenstrauss is narrower than it looks. The JL lemma (1984) guarantees that a random linear projection to O(ε⁻² log n) dimensions preserves all pairwise Euclidean distances of a fixed set of n points within a factor 1 ± ε. That is all it guarantees. It does not describe a PCA spectrum (PCA is data-adaptive; JL is not), it does not bound the dimension a task needs, and — critically — a 1 ± ε distance distortion can still reorder near-neighbours (two candidates 0.02 apart can swap under a 10% distortion), and retrieval lives on that order. Use JL as a ceiling on random-projection index dimension, not as a floor on meaning.

Three operations people call “truncation”

They are not the same, and each produces a new embedding space with its own identity (Chapter 17):

Post-hoc PCA of an ordinary model and prefix truncation of a Matryoshka model are different operations with different curves — the dimensionality report treats each as a distinct method.

Compressibility is task-specific

The clearest evidence comes from Tsukagoshi & Sasano (2025), who measured intrinsic dimension and compression tolerance across eight embedding models on four MTEB task families. The intrinsic dimension is not one number for a model — it depends on what the embeddings are being asked to do:

task family        TwoNN intrinsic dimension (across 8 models)
classification            22 – 37
clustering                11 – 17
retrieval  (queries)      32 – 51
retrieval  (passages)     ~35
semantic similarity       34 – 42

Classification and clustering embeddings sit on a lower-dimensional, more anisotropic manifold and tolerate compression to under 1% of the nominal dimension. Retrieval and semantic-similarity embeddings occupy more of the space and degrade earlier. This is the same model, the same weights, the same singular-value spectrum — the compressibility changes because the task changes which directions matter.

Geometry can suggest redundancy. Only a task-preservation experiment can authorize compression.

Demonstration: RELATE’s real dimensionality

MEASURED on RELATE v0.1, Wave 2 rows 2.2–2.7 — artifacts under experiments/embeddings-from-first-principles/wave2/artifacts/. Primary model all-mpnet-base-v2 (768-d); STS uses a RELATE-native relation→similarity proxy, not human labels.

Embed all 1,173 RELATE items with all-mpnet-base-v2 and measure the space’s real size (row 2.2):

nominal dimension              768
95%-of-variance dimension      205
participation ratio             67
effective rank (spectral)      387
intrinsic dimension (MLE, k=10)  6.8
intrinsic dimension (TwoNN)     4.2

Five different “sizes,” spanning two orders of magnitude — and the two intrinsic-dimension estimators land at single digits, lower even than the ~10–40 the representation-manifold literature reports (RELATE’s templated items sit on unusually flat local patches).

Now the PCA-truncation sweep, scored per task (row 2.3), with the knee marked at the smallest d still within 5% of full:

d kept       retrieval nDCG@10   STS ρ   clustering ARI   hard-neg margin
768               0.952          0.634       0.392            0.113
256               0.953          0.639       0.392            0.113
 64               0.948          0.634       0.432            0.122
 32               0.927          0.607       0.438            0.132
 24               0.913  ← knee  0.576       0.435            0.141
 16               0.874          0.550       0.439            0.127
  8               0.722          0.525       0.435  ← flat     0.129
knee (tol 5%)      24             32          8                8

Read the knees: 8 for clustering and hard-negative discrimination, 24 for retrieval, 32 for the STS proxy — a task-specific spread of 24. Clustering ARI actually rises as you compress (the extra 700+ dimensions are noise for k-means); the hard-negative margin is flat-to-slightly-up (there was never fine polarity/role structure in the geometry to lose).

The knee is not any single geometric quantity (row 2.4). knee ÷ TwoNN-ID runs from 1.9× (clustering) to 7.6× (STS) — not a constant. knee ÷ effective-rank is 0.02–0.08; knee ÷ 95%-variance-dim is 0.04–0.16. PCA-truncating to d = ID (~5) gives retrieval nDCG 0.72 — a 24-point drop. A linear projection to the intrinsic dimension does not reconstruct a curved manifold.

The one predictor that does better than chance (row 2.5, still a BOOK HYPOTHESIS): the effective rank of the between-relation scatter — the subspace that separates the typed relations — is 5.4, close to the relation-separation knee (8) and far better than the full effective rank (387, ~48× too big). But the analogous retrieval-restricted rank (75) overshoots the retrieval knee (24) by 3×. A task-restricted rank is the most promising knee predictor found so far; it is not yet a law.

Operator choice matters too (rows 2.6–2.7). Retrieval knee by compression method: post-hoc PCA 24 < random Gaussian projection 48 < Matryoshka-prefix truncation 64 (the Matryoshka arm is mxbai-embed-large-v1, a different, more anisotropic model — so this compares operators on their native models). And PCA beats random projection on both retrieval and the hard-negative margin at every d — the hypothesis that PCA would sacrifice fine structure to variance did not hold, because that fine structure was never there.

MEASURED: quality is flat, then a knee, then collapse — and the knee is task-set, not spectrum-set. It is not the intrinsic dimension, not the effective rank, not the 95%-variance dimension. Some compression is always safe (clustering and hard-negative discrimination survive d = 8); how much is a per-task retention curve you have to run.

What this chapter establishes and what it does not

Establishes: nominal dimension is a spec, not a measurement; effective rank, participation ratio, and intrinsic dimension are distinct, computable quantities; the gap between them is large and explained; large redundancy is typical, so some compression is safe.

Does not establish: a universal target dimension (measured on RELATE it ranged from d = 8 for clustering to d = 32 for the STS proxy on one model); that low intrinsic dimension is always bad (tight domains legitimately have low ID); a spectrum ratio that predicts the knee (row 2.4 — none does; the knee/ID ratio is itself task-dependent, 1.9×–7.6×). It establishes that you must compute these numbers and run a per-task retention curve before choosing a storage dimension, and that post-hoc PCA is a stronger compressor than random projection or (on RELATE) Matryoshka-prefix truncation.

Lab 7: measure the real size of your space

PROPOSED, not executed.

Setup. Embed 2,000+ items. Center the matrix. Run SVD.

Task.

1. Plot the singular-value spectrum (log scale).
2. Compute effective rank, participation ratio, stable rank.
3. Estimate intrinsic dimension with two estimators (TwoNN and MLE); compare — they can disagree by 2×, and the estimate depends on which subset of the data you feed them.
4. PCA-truncation sweep: d ∈ {full, 256, 128, effRank, 2·ID, ID, ID/2}; score each task separately — retrieval Recall@10/MRR, similarity Spearman, clustering ARI vs full-dim, hard-negative agreement.

Success criterion. A retention curve per task, the knee of each, and the ratio of each knee to (effective rank, TwoNN-ID, 2×ID, 95%-variance d). On RELATE v0.1 with mpnet-base the answer was: no single geometric ratio predicts the knee across tasks — the knee/ID ratio alone spans 1.9×–7.6× (row 2.4). The one lead worth chasing is the effective rank of the task-restricted subspace (row 2.5). Repeat on a second RELATE domain split and a second model.

Companion component: the dimensionality report

dimensionality_report(space, corpus):
  nominal_d:          int
  effective_rank:     float
  participation_ratio: float
  intrinsic_dim:      float  (+ estimator, + CI)
  spectrum:           singular values
  safe_truncation:    { task: d_within_tolerance }     # per task, NOT one number
                      # e.g. {retrieval: 256, clustering: 96, sts: 300}  — from a measured retention curve
  binding:            { corpus_hash, space_hash, tolerance,
                        method: pca_posthoc | whitening_posthoc | matryoshka_prefix | learned_projection }

A single “safe dimension” is a category error: the d that holds retrieval within tolerance can destroy signal a clustering, classification, or hard-negative task needs (Tsukagoshi & Sasano, 2025), and the four methods above have four different degradation curves. The report therefore carries a table keyed by task, populated from a measured retention curve, bound to a specific corpus and space. The Observatory picks the storage dimension from the tightest task it must serve, registers the truncated space as a new derived space (Chapter 17) with its own space_hash and its own calibration, and flags a model whose effective rank is a small fraction of its nominal d as “low capacity utilization — verify it is not just inefficient.”

Failure modes

• Reporting nominal dimension as capacity. “It’s a 1,536-dim model” says nothing about how much it uses.
• Confusing effective rank with intrinsic dimension. Global spectrum vs. local manifold — different numbers, different uses.
• Truncating without a per-task retention curve. The safe dimension for retrieval can be far too small for clustering or hard-negative discrimination, and vice versa. The knee is task-set, not spectrum-set — and it is not the intrinsic-dimension estimate.
• Citing Johnson–Lindenstrauss to justify a PCA dimension. JL is about random projections and pairwise distances; PCA is data-adaptive and JL says nothing about it, nor about near-neighbour ordering.
• Treating a truncated index as “the same space, smaller.” It is a new derived space (Chapter 17) — new space_hash, new calibration.
• Assuming a bigger model uses its dimensions. Sometimes the opposite; measure.

What this chapter established

• Six “dimension” quantities: nominal, rank, effective rank, participation ratio, intrinsic dimension, stable rank — and what each measures.
• The gap between nominal and effective/intrinsic dimension is large, and explained by correlation, anisotropy, thin manifolds, and no training pressure to spread.
• Large redundancy is typical, so some compression is safe — but the safe amount is task-specific (classification and clustering compress far harder than retrieval and similarity on identical embeddings), the knee of the retention curve is not the intrinsic-dimension estimate, and it must be measured per task.
• Post-hoc PCA, Matryoshka prefix truncation, and learned compression are three different operations; each produces a new derived space.
• Johnson–Lindenstrauss guarantees only that a random projection preserves pairwise distances of a fixed point set — not a PCA spectrum, not task structure, not near-neighbour ordering.
• The dimensionality report, with a safe_truncation table keyed by task (from a measured retention curve) and a method field, that the Observatory uses to choose storage size.
• Governing rule: geometry can suggest redundancy; only a task-preservation experiment can authorize compression.

Next

We have measured how many directions the space uses. The next chapter measures the shape of how it uses them — anisotropy, concentration, dominant directions, covariance structure — and compares what geometry several real models impose on the identical corpus.