Embeddings From First Principles
Chapter 01 of 26

What Is an Embedding?

By AI, Embeddings

Concepts

CHAPTER 01 — WHAT IS AN EMBEDDING?

PART I — A VECTOR IS NOT MEANING

PURPOSE

Open the book by dismantling the “an embedding captures meaning” mental model and replacing it with a precise one: an embedding is the output of a specific learned transformation under a specific objective. Introduce the RELATE corpus and the space record, the two artifacts reused for the rest of the book.

CENTRAL QUESTION

What is actually in the array of numbers an embedding model returns, and what put it there?

UNIQUE CLAIM

An embedding does not contain meaning; it is a representation produced by a learned transformation under a particular objective — so the objective determines which relationships become geometric proximity, the transformation is lossy, and the resulting geometry is conditional on model, metric, and normalization.

THE OBJECT

Representations — vectors as transformation outputs. The demonstration slice: six sentences about Dublin/Ireland/capitals where the negation of sentence A sits second-closest to A, above every unrelated sentence.

CONCEPTS INTRODUCED

Vocabulary ladder (identifier, feature, vector, representation, embedding, latent representation) — embedding defined as a learned mapping into a continuous vector space with geometrically useful structure, NOT necessarily a dimensionality reduction; latent representation is a hidden activation only counted as an embedding once we commit to reading its geometry; objective-determines-geometry (word2vec-style context prediction puts good/bad together; QA objective puts question near answer); lossy compression of vocabulary/sentence space into d floats; conditional geometry; the space record artifact.

CONCEPTS DEVELOPED / REUSED

RELATE corpus (defined here, reused every part); space record (grows into the Embedding Observatory metadata by Ch24); “aboutness vs polarity” split (paid off in Ch10 nearest-neighbor errors and Ch11 hard negatives); the book’s thesis introduced in first form — “geometry is evidence about a representation, not permission to use it” (sharpened through Parts II–VIII; operational form in Ch24: measure what survived every transformation before treating the transformed representation as equivalent).

PREREQUISITES

Comfort with vectors, dot product, and the idea of a trained model. No embeddings background.

LOCAL INVARIANTS

Name the training objective before trusting the geometry; never compare vectors across models without a space record; state the normalization; one-hot vectors are identifiers, not embeddings.

FAILURE MODES

“The embedding captured the meaning” (it captured what the objective rewarded); cross-model vector comparison; forgotten normalization; treating one-hot/identifier vectors as graded embeddings.

DIAGNOSTIC METHOD

  1. Identify the transformation and its training objective. 2. State what “close” means under that objective. 3. Record the space (model, version, dim, normalization, pooling). 4. Probe with labeled pairs to see whether “similar” means same-topic or same-claim.

RESEARCH-DERIVED IDEAS

Distributional hypothesis (Harris; Firth “you shall know a word by the company it keeps”); word2vec / GloVe as context-prediction objectives producing distributional similarity; sentence encoders trained on NLI/QA pairs producing a different “similar”; the general point that antonyms are distributionally close. Named, not cited with metadata; the Ch1 relation-cosine table is MEASURED (Wave 1 row 1.1, wave1/artifacts/relation-cosine-by-type.json).

EXPERIMENT / LAB

Lab 1 (PROPOSED): 20 pairs across 5 categories (paraphrase, negation, same-topic/different-claim, entity-overlap-only, unrelated); predict then measure mean cosine per category; answer in one paragraph whether the model’s “similar” means topic, claim, or between.

COMPANION COMPONENT

The space record — model, version, dimension, normalization, pooling, objective, observed meaning of “similar.” First brick of the Embedding Observatory.

READER OUTCOME

Reader can state, for a given encoder, what transformation and objective produced its vectors and what relationship its geometry actually encodes — and never again say an embedding “contains” meaning.

DEPENDENCIES

None (opening chapter).

FORWARD BRIDGE

Ch2 “Meaning Becomes Geometry” — once text is a point in a space, semantic questions become geometric ones; build a tiny space by hand and find where the translation leaks.

ANTI-CLAIMS / LIMITS

Does not claim embeddings are unreliable or that similarity is useless; does not claim any specific relation is or isn’t captured by any specific model — those are per-model empirical questions the book measures.

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Part I — A Vector Is Not Meaning

Three words and a list of numbers

Take three words:

cat
dog
airplane

Turn them into numbers. Any embedding API will do it. You get three arrays, each maybe 384 or 768 or 1,536 floats long:

cat       [ 0.021, -0.114,  0.062, ... ]
dog       [ 0.019, -0.098,  0.071, ... ]
airplane  [-0.087,  0.203, -0.041, ... ]

Compute the angle between cat and dog. It is small. Compute the angle between cat and airplane. It is larger. Something about “cats and dogs are both pets” appears to have survived the trip into number-space.

It is tempting to stop here and say: the embedding captured the meaning of the words.

That sentence is the first thing this book takes apart.

What is actually in that array of numbers, and what put it there?

The vocabulary we will keep separate

Five words get used interchangeably in practice. The book keeps them apart because the differences matter later.

  • Identifier. An arbitrary label. Token ID 4021 for cat. It supports equality and nothing else — 4021 is not “closer to” 4022 in any meaningful way. One-hot vectors are identifiers wearing a vector costume: every pair is equidistant.
  • Feature. A measured, named property. has_fur = 1, can_fly = 0, word_length = 3. Features are interpretable by construction; a human decided what each slot means.
  • Vector. Any element of a vector space: a list of numbers you can add, scale, and take dot products of. A vector carries geometric structure but no inherent semantics.
  • Representation. A vector produced from an input by some transformation, chosen so that geometric operations on the vector stand in for operations on the input. The transformation is the point.
  • Embedding. A representation that maps objects into a continuous vector space, learned so that a chosen relationship becomes geometrically useful structure — usually proximity. An embedding need not reduce dimensionality; a one-hot input mapped to a dense vector of the same length is still an embedding. What matters is that the space is continuous, the mapping is learned, and the geometry is meant to be used.
  • Latent representation. An intermediate activation inside a larger model — a hidden layer’s output — that can be used as an embedding. Not every hidden activation is one in the operational sense this book uses: it becomes an embedding only when we commit to reading its geometry as a stand-in for relationships between inputs.
    flowchart LR
    ID["identifier — arbitrary label; supports equality only (token ID 4021)"] --> FE["feature — measured, named property (has_fur=1); interpretable by construction"]
    FE --> VE["vector — element of a vector space; add / scale / dot product; geometry, no semantics"]
    VE --> RE["representation — a vector from an input via a transformation, so geometry stands in for input operations"]
    RE --> EM["embedding — a learned representation into a continuous space where a chosen relationship becomes usable geometry"]
    EM --> LR["latent representation — an intermediate activation read as an embedding once we commit to its geometry"]

The move from identifier to embedding is the move from “these are different” to “these differ in graded, structured ways.”

An embedding does not contain meaning

Here is the claim the rest of the book leans on.

An embedding does not contain meaning. It is a representation produced by a learned transformation under a particular objective.

Three consequences follow immediately.

The objective decides what is preserved. A model trained to predict neighboring words builds a space where distributionally similar words are close — so good and bad end up near each other, because they appear in nearly identical contexts. A model trained on question–answer pairs builds a space where a question is close to its answer, which are distributionally dissimilar. Same input text, different objective, different geometry, opposite notion of “similar.”

The transformation is lossy. Reducing a 50,000-word vocabulary, or the space of all English sentences, to 768 numbers throws information away. What it keeps is whatever reduced the training loss. Everything else — details the objective did not reward — is compressed or discarded.

The geometry is conditional. “cat is near dog” is a fact about this model’s output space under this metric. Change the model, the metric, or the normalization and the statement can change.

None of this makes embeddings less useful. It makes them a tool with a spec sheet instead of a magic trick.

This is the first form of an idea the book keeps sharpening: geometry is evidence about a representation, not permission to use it. A small angle between two vectors is a fact about where a particular transformation placed them. Whether that fact licenses a decision — “these are duplicates,” “this passage answers the query,” “this translated vector is as good as a native one” — is a separate question, and the answer is a measurement, not an assumption.

Demonstration: the RELATE corpus

MEASURED on RELATE v0.1 (corpus_hash 8cad6816…9589b3), Wave 1 row 1.1 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/relation-cosine-by-type.json. Five sentence encoders; the table shows bge-large-en-v1.5.

Throughout the book we work with one designed dataset, the RELATE corpus: 1,173 short text items with 1,181 pairs labeled by relationship —

equivalent    paraphrase    entailment    topic-related   entity-related
partial-support   contradiction   negation   temporal-mismatch   unrelated   hard-negative

Embed every item and take the mean cosine similarity within each typed pair:

relation             mean cosine (bge-large)   what it should be
equivalent                 0.96                 high (same claim)
relation-swap              0.99                 LOW  ("Acme acquired Beta" vs "Beta acquired Acme")
partial-support            0.90                 mid
paraphrase                 0.89                 high
entailment                 0.85                 mid-high
negation                   0.83                 LOW  (opposite claim)
contradiction              0.83                 LOW
topic-related              0.77                 mid
temporal-mismatch          0.74                 LOW  (right relation, wrong year)
entity-related             0.72                 low-mid
unrelated                  0.36                 low

Read the relations that should be low. relation-swap is the highest-cosine relation of all except equivalent — reversing who acquired whom barely moves the vector. negation (0.83) sits right on top of contradiction (0.83) and only 0.06 below paraphrase. Across the five models the paraphrase-minus-negation gap runs from +0.06 (bge-large, mpnet) down to −0.06 for all-MiniLM-L6-v2 — where the negation is more similar to the source than the paraphrase is. Only unrelated — no shared topic, no shared entity — separates cleanly.

MEASURED: the geometry encodes aboutness strongly (paraphrase, topic-related, entity-related all land well above unrelated) and assertion — polarity, argument order, time — barely at all. “X did Y” and “X did not do Y” occupy nearly the same point.

That is not a bug. It is the training objective showing through: under context-prediction and retrieval objectives, a sentence and its negation appear in nearly the same contexts.

What this chapter establishes and what it does not

Establishes: an embedding is the output of a specific learned transformation; the training objective determines which relationships become geometric proximity; the transformation is lossy and its geometry is conditional on model, metric, and normalization.

Does not establish: that embeddings are unreliable, that similarity is useless, or that any particular relationship (topic, polarity, entailment) is or is not captured by a given model. Those are empirical questions the book answers with measurements, per model, per relation.

Lab 1: what does your encoder think “similar” means?

PROPOSED, not executed. The deliverable is your own table.

Setup. Pick one embedding model. Take 20 sentence pairs, 4 from each of: paraphrase, negation, same-topic-different-claim, entity-overlap-only, unrelated. Record the model name, version, output dimension, and whether you normalize.

Task.

  1. Predict, before embedding, the mean cosine similarity you expect for each of the five categories.
  2. Embed all pairs; compute cosine similarity.
  3. Fill the table.
Category Predicted mean cos Observed mean cos Rank vs. unrelated
paraphrase ___ ___ ___
negation ___ ___ ___
same-topic / different-claim ___ ___ ___
entity-overlap only ___ ___ ___
unrelated ___ ___ ___

Success criterion. A one-paragraph answer to: for this model, does “similar” mean same-topic, same-claim, or something in between? If negation scores near paraphrase, your model encodes aboutness, not assertion — write that down; Chapters 10 and 11 return to it.

Companion component: the space record

The Embedding Observatory we build by Chapter 24 starts here, with the smallest possible artifact: a record of which transformation produced a vector.

space_record:
  model:          <name>
  version:        <string or commit>
  dimension:      <int>
  normalization:  <none | l2 | whitened>
  pooling:        <cls | mean | last>
  objective:      <what the model was trained to make close>
  notes:          <what "similar" appears to mean, from Lab 1>

Every vector in the book travels with one of these. A vector without a space record is a list of numbers whose meaning you have chosen to forget.

Failure modes

  • “The embedding captured the meaning.” It captured whatever the objective rewarded. Name the objective before trusting the geometry.
  • Comparing vectors from different models. Two arrays of the same length from different encoders are not in the same space (Chapter 16). The dot product is defined; the interpretation is not.
  • Forgetting normalization. Cosine and dot product agree only when vectors are unit length. Half the “why are my scores weird” problems are an un-normalized vector (Chapter 4).
  • Treating one-hot vectors as embeddings. Equidistant identifiers carry no graded structure; nearest-neighbor search over them is exact-match search.

What this chapter established

  • The vocabulary ladder: identifier → feature → vector → representation → embedding → latent representation.
  • The load-bearing claim: an embedding is a learned transformation’s output under an objective, not a container of meaning.
  • Three consequences: the objective decides what is preserved, the transformation is lossy, the geometry is conditional.
  • The RELATE corpus and the space record — the two artifacts the whole book reuses.
  • A demonstration where the negation of a sentence sits almost on top of the sentence, because the objective encodes topic more strongly than polarity.

Next

If an embedding is a transformation into a space, then semantic questions become geometric questions in that space — how close, which direction, how dense. The next chapter builds a tiny space by hand and watches meaning turn into coordinates, distance, and angle, and marks exactly where that translation starts to leak.