Part II — Inside the Space

The question a vector cannot answer alone

Hand someone a single embedding vector and ask what it means. They cannot say. Hand them the vector and its ten nearest neighbors and they can usually tell you the topic, the register, and roughly what it is.

Meaning, operationally, is locality: a vector is characterized by what it is near. So the honest object of study is not the point — it is the neighborhood.

How uniform is local structure across an embedding space, and where is it untrustworthy?

The vocabulary of local structure

• k-nearest neighbors. The k closest points to a query. The atomic operation of retrieval, clustering, and deduplication.
• Local density. How tightly packed a region is. Distance-to-10th-neighbor is a cheap estimate. Dense regions: common topics. Sparse regions: rare or unusual content.
• Cluster. A connected dense region. May or may not correspond to a human category.
• Boundary. Where two clusters meet. Items here have mixed neighborhoods and unstable rankings.
• Hub. A point that appears in an unusually large number of other points’ neighbor lists. Hubs are a high-dimensional pathology, not a sign of importance.
• Outlier / anti-hub. A point that is in nobody’s neighbor list. Often short, generic, or malformed text.

Hubness: the defect that looks like relevance

In high-dimensional spaces, the distribution of “how often is point p someone’s nearest neighbor” becomes badly skewed. A few points become nearest neighbor to hundreds of queries regardless of the query’s topic.

low-dimensional space:   every point is NN to ~k others, roughly uniform
high-dimensional space:   a few hubs are NN to 100s; many points are NN to 0

Why it matters: a hub near the top of every result list looks like a great match and is actually a geometric artifact. Mitigations exist — mutual-kNN, local scaling, cross-domain similarity normalization, centering — and they belong in the retrieval policy (Chapter 12). Here, the point is to measure hubness before trusting raw kNN.

The manifold assumption, and where it breaks

The comforting story: embeddings of real text lie on a low-dimensional curved surface (a manifold) inside the high-dimensional space, and distances along that surface track meaning. Often roughly true. Where it breaks:

• Between clusters, there is no surface — the “distance” across a gap between two topic clusters is not a meaningful semantic quantity, just Euclidean space.
• Rare content sits off the manifold — few similar examples means the local surface is poorly defined.
• Different regions have different intrinsic dimension — a tight cluster of near-duplicates is locally 1–2 dimensional; a diverse region is locally much higher (Chapter 7).

The visualization trap, revisited with teeth

t-SNE and UMAP optimize to preserve local neighbor relationships and deliberately distort global distance. So:

what a UMAP plot shows well:   which points are in a tight local group
what it shows badly:           how far apart two groups are
                               whether a "gap" is real or an artifact of perplexity/n_neighbors
                               density (both methods equalize it by default)

A demonstration to run once and remember: take three well-separated Gaussian blobs in 50D, plus uniform noise. UMAP will often render the noise as its own tidy “cluster” and place the blobs at arbitrary relative distances. The plot has structure the data does not.

Demonstration: RELATE neighborhoods are lumpy

MEASURED on RELATE v0.1, Wave 1 row 1.5 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/hubness.json.

Embed all 1,173 RELATE items. Compute full-space 10-NN for every item (mean in-degree is 10 by construction).

model               max in-degree   in-degree skew   Recall@10   Recall@10, top-1% hubs removed
all-mpnet-base-v2         31             +0.6           1.000            0.991
all-MiniLM-L6-v2          37             +0.7           0.993            0.990
mxbai-embed-large-v1      35             +1.1           1.000            0.996
bge-small-en-v1.5         45             +1.5           0.971            0.968
bge-large-en-v1.5         49             +1.9           1.000            0.996

The in-degree distribution is right-skewed for every model — a handful of items sit in 3–5× as many neighbor lists as the average, and the skew grows with the model’s anisotropy (Chapter 5). Inspect the hubs: they are the short, generic sentences.

MEASURED: hubness is real and structural. But on a 1,173-item index it does not cost retrieval — removing the top-1% hubs leaves Recall@10 essentially unchanged (and slightly lower, because a few hubs were themselves correct answers). The “hubs displace correct answers” effect the literature reports is a property of much larger indexes; RELATE shows the geometry, not yet the damage. The space is still not a uniform fabric — crowded districts, empty lots, a few corners everyone passes through — and averaged metrics hide all of it.

What this chapter establishes and what it does not

Establishes: local structure is non-uniform; hubness is a measurable high-dimensional artifact that inflates apparent relevance; the manifold assumption holds within clusters and fails between them and for rare content; local intrinsic dimension varies.

Does not establish: that hubness always hurts a given application (measure it), or that any clustering corresponds to your categories. It establishes that raw kNN quality varies by region and that you should know your space’s hub list.

Lab 6: map the local structure

PROPOSED, not executed.

Setup. Embed 1,000+ items. Compute full-space k-NN (k = 10) for all.

Task.

1. Hubness. Histogram of “in-degree” (how many neighbor lists each item appears in). Report skew, list the top 10 hubs, read them.
2. Anti-hubs. Fraction with in-degree 0. Read a sample.
3. Density. Distance-to-kth-neighbor per item; ratio of 90th to 10th percentile.
4. Boundaries. Items whose 10 neighbors span ≥3 labels.
5. Intervention. Drop the top hubs from the index; re-score retrieval.

Success criterion. A one-page “neighborhood map” of your space: where kNN is trustworthy, where it is not, and the hub list to watch.

Companion component: the neighborhood report

neighborhood_report(space):
  hub_list:            top-k items by in-degree
  anti_hub_fraction:   float
  density_map:         per-item distance-to-kth-neighbor
  boundary_items:      ids with mixed-label neighborhoods
  local_id_estimate:   intrinsic dimension per region (from Ch7)

The Observatory attaches this to every indexed space and surfaces the hub list in retrieval diagnostics.

Failure modes

• Trusting raw kNN uniformly. Quality is regional. Hubs and sparse zones need different handling.
• Reading hubs as popular/important. They are geometric sinks.
• Believing a UMAP gap. Change n_neighbors/perplexity and see if it survives.
• Assuming density = relevance. Dense regions are common topics, not better answers.

What this chapter established

• The unit of meaning is the neighborhood, not the point.
• Vocabulary: kNN, local density, cluster, boundary, hub, anti-hub.
• Hubness is a measurable high-dimensional artifact; a few points become everyone’s neighbor (in-degree skew grows with anisotropy). On the 1,173-item RELATE index removing the top-1% hubs did not change Recall@10 — the “hubs displace correct answers” damage is an effect of much larger indexes; RELATE shows the geometry, not yet the cost.
• The manifold assumption is local: fine within clusters, meaningless between them, weak for rare content; local intrinsic dimension varies.
• The neighborhood report: hub list, anti-hub fraction, density map, boundary items — attached to every indexed space.

Next

Chapter 5 said the basis is arbitrary; Chapter 6 said local dimension varies. Both point at one question: how many dimensions does the representation actually use? The next chapter measures intrinsic dimension, effective rank, and the singular-value spectrum — and finds a 1,536-dimensional vector living in far fewer.