Embeddings on Programmer.ie: Modern AI programming

What Is an Embedding?

Mon, 07 Sep 2026 09:00:00 +0000

Part I — A Vector Is Not Meaning

Three words and a list of numbers

Take three words:

cat
dog
airplane

Turn them into numbers. Any embedding API will do it. You get three arrays, each maybe 384 or 768 or 1,536 floats long:

cat       [ 0.021, -0.114,  0.062, ... ]
dog       [ 0.019, -0.098,  0.071, ... ]
airplane  [-0.087,  0.203, -0.041, ... ]

Compute the angle between cat and dog. It is small. Compute the angle between cat and airplane. It is larger. Something about “cats and dogs are both pets” appears to have survived the trip into number-space.

Meaning Becomes Geometry

Mon, 07 Sep 2026 09:10:00 +0000

Part I — A Vector Is Not Meaning

A space you can draw

Give six words two coordinates each, by hand:

              (x = royalty/power,  y = gender: -1 female … +1 male)
king      ( 0.9,  0.8)
queen     ( 0.9, -0.8)
man       ( 0.1,  0.9)
woman     ( 0.1, -0.9)
apple     (-0.8,  0.0)
orange    (-0.8,  0.1)

Now every semantic question has a geometric form:

Are king and queen related? → they are close (distance 1.6, mostly along y).
What distinguishes king from queen? → a direction: king − queen ≈ (0, 1.6), the “gender axis.”
Is apple like king? → the angle between them is wide; they point different ways.
Which words form a group? → {apple, orange} cluster tightly, far from the royalty cluster.

This is the whole promise of embeddings in one picture:

Learning an Embedding Space

Mon, 07 Sep 2026 09:20:00 +0000

Part I — A Vector Is Not Meaning

Counting words in a tiny corpus

Six sentences:

The cat drinks milk
The dog drinks water
The kitten drinks milk
The puppy drinks water
The cat chases the mouse
The dog chases the cat

Build a co-occurrence matrix: for each word, count how often each other word appears within one step of it.

          cat  dog  kitten  puppy  milk  water  drinks  chases  mouse
cat        -    1     0      0      0     0      1       2       0
dog        1    -     0      0      0     0      1       1       0
kitten     0    0     -      0      1     0      1       0       0
puppy      0    0     0      -      0     1      1       0       0
milk       0    0     1      0      -     0      2       0       0
water      0    0     0      1      0     -      2       0       0

Each row is already a crude embedding: kitten and milk co-occur; puppy and water co-occur. The rows for cat and dog look alike because both chase and drink.

Similarity Is a Decision

Mon, 07 Sep 2026 09:30:00 +0000

Part I — A Vector Is Not Meaning

Two vectors, four answers

Here are two document vectors (3D, for legibility):

x = ( 2.0, 0.0, 0.0 )      a short doc, one strong topic
y = ( 6.0, 0.1, 0.0 )      a long doc, same topic, more of it
z = ( 0.0, 2.0, 0.0 )      a short doc, different topic

Ask “is x more similar to y or to z?” and the metric answers for you:

Dimensions Do Not Mean What You Think

Mon, 07 Sep 2026 10:00:00 +0000

Part II — Inside the Space

What does dimension 173 mean?

Pull the 173rd coordinate of every vector in your corpus and sort. You get a list of documents ordered by… something. Occasionally a coordinate is weakly interpretable — “this one seems higher for questions” — but usually the sorted list has no human-nameable theme. Dimension 173 does not mean “formality” or “sentiment” or “is about sports.”

And yet the vector works. Similarity search returns sensible results. The structure is there; it is just not living in the individual axes.

Neighborhoods and Manifolds

Mon, 07 Sep 2026 10:10:00 +0000

Part II — Inside the Space

The question a vector cannot answer alone

Hand someone a single embedding vector and ask what it means. They cannot say. Hand them the vector and its ten nearest neighbors and they can usually tell you the topic, the register, and roughly what it is.

Meaning, operationally, is locality: a vector is characterized by what it is near. So the honest object of study is not the point — it is the neighborhood.

How Many Dimensions Does Meaning Need?

Mon, 07 Sep 2026 10:20:00 +0000

Part II — Inside the Space

A 1,536-number vector that isn’t

Your embedding model outputs 1,536 floats. Take 50,000 of its vectors, stack them, run SVD, and plot the singular values. Typically:

first ~20 singular values:     large, falling fast
next ~150:                     moderate, gentle slope
remaining ~1,350:              tiny, near the noise floor

Reconstruct every vector from just the top 200 components. Re-run retrieval. Recall@10 barely moves. The other ~1,300 dimensions were carrying almost no variance that mattered for the task.

The Shape of an Embedding Space

Mon, 07 Sep 2026 10:30:00 +0000

Part II — Inside the Space

Same corpus, three shapes

Embed the same 1,173 RELATE items with five models. Do not look at task scores yet. Look at the shape (measured — Wave 1 row 1.4, experiments/embeddings-from-first-principles/wave1/artifacts/anisotropy.json):

                          MiniLM-L6   mpnet-base   mxbai-large   bge-large   bge-small
dimension                     384         768         1024          1024        384
mean random-pair cosine       0.06        0.08         0.34          0.40        0.45
top singular-value share      0.07        0.08         0.09          0.09        0.10
effective rank                 259         387          425           434         271
distance concentration*       0.08        0.09         0.09          0.11        0.10

*ratio of standard deviation to mean of pairwise distances — smaller means “everything is roughly equidistant.”

From Similarity to Search

Mon, 07 Sep 2026 11:00:00 +0000

Part III — Retrieval Is an Experiment

Retrieval in four lines

query_vec = embed(query)                       # one vector
scores    = corpus_vecs @ query_vec            # one dot product per document
order     = np.argsort(-scores)                # sort, descending
results   = [corpus[i] for i in order[:k]]     # take the top k

That is the entire primitive. Everything a vector database adds — indexes, quantization, sharding, filtering — is an optimization or an operational convenience around these four lines. Understand them completely before adding anything.

What exactly is a retrieval system deciding, and which of its parameters change the answer rather than just the speed?

The Nearest Neighbor Can Be Wrong

Mon, 07 Sep 2026 11:10:00 +0000

Part III — Retrieval Is an Experiment

The result that is closest and wrong

Query:

Is Dublin the capital of Ireland?

Top retrieved passage, cosine 0.91:

Dublin is not, and has never been, the capital of Ireland — that
distinction belongs to the older seat of government at Tara.

Fluent, on-topic, confidently phrased, geometrically the closest thing in the corpus — and false. A model handed this passage as context may repeat its claim. The retrieval system did its job perfectly: it found the nearest vector. “Nearest” was not “correct.”

Hard Negatives

Mon, 07 Sep 2026 11:20:00 +0000

Part III — Retrieval Is an Experiment

Two benchmarks, same model, opposite verdicts

Benchmark E (easy negatives):
  positive: the correct answer
  negatives: 20 passages drawn at random from the corpus
  → model scores Recall@1 = 0.94

Benchmark H (hard negatives):
  positive: the correct answer
  negatives: the 20 passages most similar to the query that are NOT correct
  → same model scores Recall@1 = 0.55

Nothing changed but the negatives. The easy benchmark asked “can the model tell the answer from an unrelated paragraph?” The hard benchmark asked “can it tell the answer from the 20 things that look most like the answer?” Those are different questions, and only the second is the one production faces.

Retrieval Is a Policy

Mon, 07 Sep 2026 11:30:00 +0000

Part III — Retrieval Is an Experiment

“Just retrieve the relevant documents”

There is no such operation. What actually runs is a pipeline, and every stage has parameters that change the output:

    flowchart TD
    Q[query] --> QT["query transform — raw / rewrite / expand / HyDE / decompose"]
    QT --> RE["representation — which model, normalization, chunking"]
    RE --> CG["candidate generation — ANN index, nprobe / ef, pre-filters"]
    CG --> SI["similarity — metric, hybrid dense+sparse weighting"]
    SI --> TH["threshold — minimum score to survive"]
    TH --> RK["ranking — rerank top-N with a cross-encoder?"]
    RK --> DV["diversity / dedup — MMR, cluster collapse"]
    DV --> TK["top-k — how many survive"]
    TK --> CA["context assembly — order, dedup, budget, citations"]
    CA --> M["what the model actually sees = the system's memory"]

Retrieval does not return “the relevant documents.” It returns whatever this specific chain of decisions produces — and that is the system’s memory.

How Do You Evaluate an Embedding?

Mon, 07 Sep 2026 12:00:00 +0000

Part IV — Measuring the Representation

The leaderboard model that lost

A team picks the top model on a public embedding leaderboard. It scores well on 50-plus tasks. In their product — retrieval over dense technical documentation with heavy entity and version-number queries — it underperforms a smaller, older model. Nothing was misconfigured. The benchmark measured a population of tasks; the product is one task, and not one the benchmark weighted heavily.

Calibration

Mon, 07 Sep 2026 12:10:00 +0000

Part IV — Measuring the Representation

What is 0.81?

A pipeline decides two documents are “duplicates” if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Then:

distribution of cosine for KNOWN duplicate pairs:      mean 0.79, sd 0.09
distribution of cosine for KNOWN non-duplicate pairs:  mean 0.71, sd 0.11

The threshold sits inside both distributions. A true duplicate at 0.78 is rejected; an unrelated pair at 0.82 is accepted. The number 0.81 does not carry its own meaning — it is a point on two overlapping bell curves, and without those curves it is noise.

Is Similarity One-Dimensional?

Mon, 07 Sep 2026 12:20:00 +0000

Part IV — Measuring the Representation

Two pairs, same cosine, different situations

Pair 1:  cos = 0.78
  the query's nearest neighbor scores 0.78; the 2nd scores 0.44
  the region is sparse; the ranking is stable under paraphrase
  → a confident, isolated match

Pair 2:  cos = 0.78
  the nearest neighbor scores 0.78; the 2nd, 3rd, 4th score 0.77, 0.76, 0.76
  the region is a dense hub; re-embedding the query with a typo flips the top result
  → a coin toss dressed as a match

The scalar is identical. The situations are not. Everything that distinguishes them is geometric information the cosine threw away.

Change the Model, Change the Universe

Mon, 07 Sep 2026 13:00:00 +0000

Part V — Embedding Spaces Are Not Universal

The same sentence, three universes

Embed one sentence with three models:

model A (384-d):  [ ... ]
model B (768-d):  [ ... ]
model C (768-d):  [ ... ]

A and B differ in length, so nobody expects to compare them coordinate-wise. But B and C are both 768-dimensional. Line up their vectors and compute the cosine between B’s vector and C’s vector for the same sentence:

Versioning the Space

Mon, 07 Sep 2026 13:10:00 +0000

Part V — Embedding Spaces Are Not Universal

The upgrade that broke search quietly

Production runs embedding model v1. A better v2 ships. Someone updates the client library. New documents get v2 vectors; the 10 million existing vectors are still v1. Nothing errors. Queries are embedded with v2 and compared against a mix of v1 and v2 vectors.

Recall drops a few points. Nobody notices for a month, because the system still returns ten results and they still look plausible. The v1 and v2 vectors are in different universes (Chapter 16), and half the index is now in the wrong one.

Can One Embedding Space Be Translated Into Another?

Mon, 07 Sep 2026 14:00:00 +0000

Part VI — Crossing Embedding Spaces

The question, stated carefully

We have two encoders, E_A and E_B, and a set of objects x. Each object has two representations: E_A(x) in space A, E_B(x) in space B. Chapter 16 established these live in unrelated coordinate systems.

The question:

Does there exist a map T such that T(E_A(x)) ≈ E_B(x) for objects x the map was never trained on — and by what standard do we judge “≈”?

Alignment

Mon, 07 Sep 2026 14:10:00 +0000

Part VI — Crossing Embedding Spaces

A family of maps, ordered by how much they can bend the space

The null map (T(x) = x). Do nothing. Always run this first. If it already scores well, your two spaces were nearly the same space to begin with (this happens for models of the same size, backbone, and objective) and any “translation” you fit is fitting noise. vec2vec’s own baseline is exactly this, and for their near-identical model pairs it reaches top-1 ≈ 1.0 — the real work is only on pairs where the null map collapses.
Orthogonal Procrustes. T is constrained to a rotation/reflection: min ‖X_A R − X_B‖² subject to RᵀR = I. Solution: R = UVᵀ from the SVD of X_Aᵀ X_B. Preserves all distances and angles within space A — it only re-orients. Cannot fix scale or shear. Best when the two spaces are “the same shape, different pose.”
Linear regression (least squares). Chapter 18. T is any linear map: rotation + scaling + shear + projection. More expressive; can overfit; does not preserve within-A geometry.
CCA (Canonical Correlation Analysis). Finds paired directions in A and B that are maximally correlated; aligns in that shared subspace. Naturally handles different dimensions and discards directions with no counterpart. Good when the two spaces share a subspace but also have private structure. Note it maximizes correlation, which is not the same as preserving retrieval rank — highly correlated shared directions can still reorder near-ties.
Relative representations (“re-express, don’t map”). Instead of fitting a map at all, replace every vector by its list of cosine similarities to a fixed set of shared anchor items (Moschella et al., 2023). That representation is invariant to rotation and rescaling by construction, so two spaces expressed this way become directly comparable — no T to fit. The cost: it needs a shared anchor set (texts embedded in both spaces — paired data by another name), and the invariance holds “under the same data and modeling choices.”
Learned nonlinear map (small MLP). T is a 2–3 layer network. Most expressive; needs the most anchors; highest overfitting risk; hardest to reason about. Use only if the linear family measurably plateaus below your target.

Map	What it can do	Overfitting risk	Best when
Null `T(x) = x`	nothing — run it first	none	the two spaces were already nearly one
Orthogonal Procrustes	rotation / reflection only; preserves within-A distances and angles	low (closed form)	“same shape, different pose”
Linear regression	rotation + scale + shear + projection	medium; can overfit	scale or shear differ; anchors plentiful
CCA	align a maximally-correlated shared subspace; drop private directions	medium	the spaces share a subspace but also have private structure
Relative representations	re-express every vector as cosines to shared anchors — no `T` fitted	n/a	you have a shared anchor set and want rotation/scale invariance for free
Nonlinear MLP	an arbitrary map	highest (~10⁶ params)	the linear family measurably plateaus below target (or unpaired optimization needs the flexibility)

The ordering is deliberate: try the most constrained map that meets your preservation target. A rotation that gets you to 0.85 retrieval agreement is better than an MLP that gets you to 0.87, because the rotation has fewer ways to be wrong on data you have not seen. The recent cross-encoder alignment work reinforces this: vec2vec’s nonlinear translator and mini-vec2vec’s linear one reach comparable coarse quality, and the linear one is far cheaper and more stable (Dar, 2025). Where nonlinearity earns its keep is not representational capacity but optimization robustness in the unpaired regime — when there are no anchors to regress on, the extra flexibility helps the alignment converge, not the final map express more.

The Embedding Bridge

Mon, 07 Sep 2026 14:20:00 +0000

Part VI — Crossing Embedding Spaces

A map is not a bridge

Chapter 19 produced a matrix W (or an MLP) that translates vectors from space A to space B. Deploying that matrix as “space A and space B are now compatible” is the mistake this chapter prevents.

A map is a function. A bridge is a map plus the record of what it was shown to preserve, for whom, under what conditions — and, crucially, an explicit scope of what it must not be used for.

Did the Bridge Preserve the Space?

Mon, 07 Sep 2026 14:30:00 +0000

Part VI — Crossing Embedding Spaces

One bridge, eight verdicts

Take the linear A→B bridge from Chapter 19. Run every preservation metric on the same held-out set:

cosine preservation (mean cos to native target)      0.74
top-1 preservation (native NN stays NN)               0.61
top-10 preservation                                   0.88
Recall@1  (vs labels, translated query)               0.58   native: 0.63
Recall@10                                             0.71   native: 0.77
MRR                                                   0.64   native: 0.70
nDCG@10                                               0.75   native: 0.81
order preservation (Kendall τ on result lists)        0.66
cluster preservation (ARI vs native clustering)       0.80
hard-negative agreement                               0.41

Ten numbers, not one. A stakeholder who asked “did the translation work?” gets a table, and the table says: coarse structure preserved, fine order degraded, hard distinctions near the source encoder’s own native performance.

Can a Smaller Representation Preserve a Larger One?

Mon, 07 Sep 2026 15:00:00 +0000

Part VII — What Survives Transformation

Two vectors for one document

full_doc      →  E(full_doc)        one vector
summary(doc)  →  E(summary(doc))    one vector

Both are single vectors of the same dimension. One was produced from 4,000 words; the other from 120. The question:

How much of the document’s representational geometry survives when the document is compressed — and is the surviving-or-not a signal we can use?

If E(summary) lands where E(full_doc) lands and behaves like it under retrieval and clustering, the summary is a faithful compression in the sense the embedding cares about. If it drifts, something was dropped — and which direction it drifted may tell us what.

From Deltas to Operators

Mon, 07 Sep 2026 15:10:00 +0000

Part VII — What Survives Transformation

The other thing a subtraction might mean

king − man + woman ≈ queen is the famous demonstration that a direction in embedding space can correspond to a semantic relation. It is also, as Chapter 2 noted, partly curated and works best locally — and it has a long list of documented problems: the offset method’s success is entangled with plain cosine-neighbourhood structure, so a “the direction transfers” result has to beat the baseline of ignoring the offset and returning the nearest neighbour of the source word (Linzen, 2016); and performance varies wildly by relation type (Rogers, Drozd & Li, 2017).

Building an Embedding Runtime

Mon, 07 Sep 2026 16:00:00 +0000

Part VIII — Embeddings Become Infrastructure

Not a RAG chatbot

The obvious capstone would be “build a RAG system.” It would also be the wrong one — a RAG system exercises maybe a third of this book and hides the rest inside a framework.

The capstone is an Embedding Observatory: a runtime whose job is to know things about representations. It ingests text, documents, and queries under one or more embedding models, and it exposes not just vectors but the geometry, neighborhoods, distributions, evaluations, calibrations, versions, and bridges that the book spent 23 chapters building tools for.

Retrieval Is Not Geometry

Wed, 09 Sep 2026 01:50:00 +0000

Research Coda — What Alignment Really Preserves

The result that looks like success

Suppose a bridge translates vectors from space A into space B. On a held-out set, the true B-space counterpart appears in the translated vector’s top ten 99.7% of the time.

That sounds finished.

Now ask a different question: does the translated vector have the same ten neighbors as the native B-space vector?

The answer can be barely 59%.

What Should a Translation Preserve?

Wed, 09 Sep 2026 02:00:00 +0000

Research Coda — What Alignment Really Preserves

“Preserve the geometry” is not a complete instruction

A bridge translates vectors from space A into space B.

The natural instinct is to say:

Preserve the source geometry while you translate.

That sounds obviously correct. If two source points are close, keep them close. If two source points are far apart, keep them far apart. Preserve pairwise cosine, distances, neighborhoods.

But Chapter 16 established something uncomfortable: different encoders do not merely rotate the same universe. They can disagree on neighborhoods, density, rank order, hard distinctions, and calibration.