Part I — A Vector Is Not Meaning

Counting words in a tiny corpus

Six sentences:

The cat drinks milk
The dog drinks water
The kitten drinks milk
The puppy drinks water
The cat chases the mouse
The dog chases the cat

Build a co-occurrence matrix: for each word, count how often each other word appears within one step of it.

          cat  dog  kitten  puppy  milk  water  drinks  chases  mouse
cat        -    1     0      0      0     0      1       2       0
dog        1    -     0      0      0     0      1       1       0
kitten     0    0     -      0      1     0      1       0       0
puppy      0    0     0      -      0     1      1       0       0
milk       0    0     1      0      -     0      2       0       0
water      0    0     0      1      0     -      2       0       0

Each row is already a crude embedding: kitten and milk co-occur; puppy and water co-occur. The rows for cat and dog look alike because both chase and drink.

Where does the “meaning” in a learned embedding come from — and what operation actually produces it?

The answer, visible in this table: the meaning comes from the corpus’s statistical structure, and the operation is compression of that structure to a low-rank form.

Three ways to compress the same counts

1. Truncated SVD of the count matrix. Factor the (log- or PPMI-weighted) co-occurrence matrix M ≈ U Σ Vᵀ and keep the top k columns of U. Each word gets a k-dimensional vector. With k = 2, cat and dog land close; milk and water land close; the two clusters separate. This is, up to weighting details, what classic count-based word vectors (LSA, GloVe, PPMI-SVD) do.

2. A tiny predictive model. Train a model to predict a word from its neighbors (or neighbors from a word). The learned input-layer weights are the embeddings. This is skip-gram / CBOW. It is known that this predictive objective is, mathematically, closely related to factorizing a shifted PMI matrix — so the “prediction vs. counting” distinction is smaller than it looks.

3. A modern encoder. A transformer trained on a next-token or contrastive objective over billions of tokens, producing contextual vectors. Same idea, vastly more capacity, plus context-dependence: bank gets different vectors in “river bank” and “savings bank.”

    flowchart TD
    C["corpus — text at scale"] --> M["co-occurrence structure — which items share contexts"]
    M --> S1["truncated SVD of the PPMI / log-weighted count matrix (LSA, GloVe, PPMI-SVD)"]
    M --> S2["a tiny predictive model — predict a word from its neighbours (skip-gram / CBOW)"]
    M --> S3["a transformer encoder — next-token or contrastive objective over billions of tokens"]
    S1 --> E["low-dimensional summary that preserves the co-occurrence structure"]
    S2 --> E
    S3 --> E
    S2 -.->|"mathematically ≈ factorizing a shifted-PMI matrix"| S1
  

All three are the same move: observe co-occurrence structure at scale, keep a low-dimensional summary that preserves it.

What “learned compression” implies

The vector is a summary of contexts, not a definition. milk is near water because they occupy the same slots in sentences, not because the model knows either is a liquid.

Rare things are represented poorly. A word or concept seen a handful of times has a noisy vector — few contexts to compress. This shows up later as unstable neighborhoods (Chapter 6) and miscalibrated similarity (Chapter 14).

The corpus’s biases become the space’s geometry. If the training text associates two concepts, the vectors will too, whether or not the association is one you want.

“Meaning” is an emergent, approximate property. It emerges from enough data and enough capacity, it is approximate, and it degrades gracefully rather than failing loudly — which is exactly what makes embedding failures hard to notice.

Demonstration: grow the RELATE geometry from counts

MEASURED on RELATE v0.1, Wave 1 row 1.2 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/ppmi-svd-relate.json.

Take the 1,173-item RELATE corpus. Build a term–term co-occurrence matrix (co-occurrence within an item), apply PPMI weighting, truncate with SVD to 100 dimensions, and use the mean of a sentence’s term vectors as its embedding. Vocabulary: 1,349 words.

relation             mean cosine (PPMI-SVD-100)
equivalent                 0.98
relation-swap              1.00
temporal-mismatch          1.00
negation                   0.95
partial-support            0.91
contradiction              0.89
paraphrase                 0.80
entailment                 0.74
topic-related              0.72
entity-related             0.64
unrelated                  0.28

The homemade space separates related from unrelated — the mean of paraphrase, topic-related, entity-related sits 0.44 above unrelated. And it confuses contradiction with paraphrase: contradiction (0.89) scores higher than paraphrase (0.80), so the gap is −0.10, inverted. negation, relation-swap, and temporal-mismatch are near 1.0 — a bag-of-term-vectors cannot represent a “not”, a role reversal, or a year at all.

MEASURED: this is the same failure shape the neural encoder showed in Chapter 1 (row 1.1) — related-vs-unrelated works, assertion-vs-aboutness does not — only cruder. Scale sharpens the compression of the easy relations; it does not change what is being compressed away.

What this chapter establishes and what it does not

Establishes: embedding vectors are low-rank summaries of co-occurrence structure; counting and prediction are closely related routes to the same summary; the properties that follow (poor rare-item vectors, inherited corpus bias, emergent-and-approximate meaning).

Does not establish: that count-based and neural embeddings are interchangeable in practice (they are not, at scale), or that any specific relation is captured — still an empirical, per-model question.

Lab 3: your own word vectors in fifty lines

PROPOSED, not executed.

Setup. Take a 50k–500k word corpus. Build a co-occurrence matrix with a window of 2–5. Apply PPMI. Truncate with SVD to k ∈ {10, 50, 200}.

Task.

1. For 10 seed words, list the 5 nearest neighbors at each k.
2. Test 5 analogies (a : b :: c : ?) with vector arithmetic.
3. Compare against a pretrained model on the same 10 seed words.

Success criterion. State how k changes the neighborhoods (too small: broad topical mush; too large: noise and rare-word artifacts) and name one analogy your homemade space gets right and one it gets wrong.

Companion component: the corpus fingerprint

Before embedding a corpus, the Observatory records what the space will be a compression of:

corpus_fingerprint:
  n_items:            <int>
  vocab_size:         <int>
  token_count:        <int>
  length_distribution: <summary>
  rare_item_fraction: <items with < N contexts>
  duplicate_fraction: <near-duplicates>
  domain_notes:       <what topics/registers dominate>

Rare-item fraction and duplicate fraction predict where the geometry will be untrustworthy later.

Failure modes

• “The model knows X.” It compressed contexts in which X appeared. It knows co-occurrence.
• Trusting rare-item vectors. Few contexts, noisy summary — flag them, do not rank them confidently.
• Ignoring corpus provenance. A space trained on web text has web text’s associations; a space trained on your support tickets has your product’s.
• Assuming bigger k is better. Past a point, extra dimensions add noise and over-fit frequency artifacts (Chapter 7).

What this chapter established

• A co-occurrence matrix is already a crude embedding; SVD/factorization compresses it to a usable one.
• Counting and predicting are closely related routes to the same low-rank summary (skip-gram ≈ shifted-PMI factorization).
• Consequences of “learned compression”: poor rare-item vectors, inherited corpus bias, emergent-and-approximate meaning that fails quietly.
• A homemade 100-dimensional RELATE space that separates related from unrelated and confuses contradiction with paraphrase — the same boundary the large model has.
• The corpus fingerprint: a record of what the space compresses, with the fields that predict later untrustworthiness.

Next

We now have spaces — handed to us and homemade. Every use of them so far has quietly assumed one metric. The next chapter makes the metric a first-class decision: cosine, dot product, Euclidean, Manhattan, normalized or not — and shows that “similar” is a choice, not a measurement.