Part VII — What Survives Transformation

The other thing a subtraction might mean

king − man + woman ≈ queen is the famous demonstration that a direction in embedding space can correspond to a semantic relation. It is also, as Chapter 2 noted, partly curated and works best locally — and it has a long list of documented problems: the offset method’s success is entangled with plain cosine-neighbourhood structure, so a “the direction transfers” result has to beat the baseline of ignoring the offset and returning the nearest neighbour of the source word (Linzen, 2016); and performance varies wildly by relation type (Rogers, Drozd & Li, 2017).

So this chapter takes the idea seriously but stops treating the difference vector as the object. Define it anyway:

Δ(x₁, x₂) = E(x₂) − E(x₁)

For a pair like (rough draft sentence, edited sentence), Δ points from “before” to “after.” The question is not “does Δ transfer?” It is:

What is the simplest operator class that represents this transformation out of sample — while leaving the geometry it should not change alone?

The operator ladder

Start from one general form and read the chapter’s separate ideas as special cases of it:

T(x) = W x + b

Two connections make this the culmination of Part VI rather than an analogy appendix:

• A delta is an affine map with W = I. Rung 1 is a special case of rung 4.
• A linear embedding bridge (Chapters 18–21) is a transformation with b = 0. Aligning two spaces and applying a semantic edit are the same operation at different rungs of the same ladder. “A bridge has a complexity” and “a transformation has a complexity” are one statement.

The hypothesis this chapter set out to test was that transformation complexity is relation-specific — that some transformations behave like a global direction, others need a source-dependent map, and some admit no reusable operator. The Transformation Wave (below) measured it on RELATE-DOC and found something sharper and more one-sided: for sentence embeddings, the operator ladder above rung 1 buys nothing. Every transformation on the test set is either identity (the embedding does not move — there is nothing to transform) or has no passing operator at any rung (the embedding moves, and no operator up to an MLP captures the move). The “middle” of the ladder — the reusable rank-r or affine direction the analogy tradition assumed — was not observed for any of the nine transformations.

Where reusable operators plausibly exist — and where they plausibly do not

Plausibly reusable (some rung clears the bar):

• Editorial transformations. “Wordy → concise,” “passive → active,” “informal → formal.” If a writing system remembers (source, target) edit pairs, an operator fitted to those could apply the same edit to a new sentence, and operator similarity could retrieve past edits of the same kind.
• Grammatical transformations. “Singular → plural,” “present → past” — highly regular; a candidate for a low rung.
• Relation directions. “Country → its capital,” “company → its CEO” — the classic analogy setting.

Plausibly not (no rung clears the bar under a reasonable collateral budget):

• Content-dependent transformations. “Summarize” depends entirely on what is being summarized; there is no single summarize-operator (this is Chapter 22’s territory).
• Anything the base model does not represent as accessible structure. If polarity is near the model’s noise floor under cosine (Chapter 10), “negate this” may have no reusable operator at any rung.
• Abstract multi-step reasoning. “Problem → solution” is not one operator.

Borrowed from model internals — and why it does not carry over

There is a large literature showing that inside a language model’s residual stream, some concepts are approximately linear directions (Park, Choe & Veitch, 2024), that a single steering vector shifts behaviour (Turner et al., 2023; Rimsky et al., 2024), and that others need an affine correction rather than a bare direction, or a low-rank weight edit (Meng et al., 2022; 2023).

That work is suggestive that the ladder is real. It is not evidence about an external sentence encoder. A model’s hidden states are shaped by the next-token objective and are read through the unembedding; a sentence embedding is a different object with a different objective and a different geometry (Chapter 1). This chapter tests the ladder on sentence embeddings directly and imports no conclusions from the internals literature — only the vocabulary of operators and the failure modes.

Knowledge graphs walked this exact ladder

Independently, the knowledge-graph embedding community climbed the same rungs, each step forced by a class of relations the previous rung could not represent:

TransE   relation = a translation:  h + r ≈ t        works for 1-to-1 relations,
                                                     breaks on 1-to-N / N-to-1 / N-to-N
TransH   project onto a relation-specific hyperplane, then translate
TransR   apply a relation-specific projection matrix, then translate   (a per-relation affine map)
RotatE   relation = a rotation in complex space       composes and inverts; captures
                                                     symmetry / antisymmetry / inversion patterns

The lesson transfers cleanly: “complex relations need a matrix, not a vector” is a result someone else already paid for. Chapter 23 asks which of our relations are which.

The operator bake-off

For a transformation r with held-out pairs, fit every rung on the same training pairs and evaluate all of them the same way.

1. Collect n pairs {(a₁, b₁), ..., (aₙ, bₙ)} of transformation r, with a template family and an entity family recorded per pair.
2. Split by template and by entity (not random rows) — held-out template and held-out entity are the transfer tests that matter.
3. Fit rungs 0–6 on the training split.
4. On the held-out split, compute the transformation preservation profile (below) for each rung.
5. Find the simplest passing operator: the lowest rung that clears the target-preservation bar ε without exceeding the collateral-distortion budget δ.

Baselines (from the analogy critiques): rung 0 (T(x) = x), a random Δ, and — crucially — ignore the operator, return the nearest neighbour of the source. A “transfer” that does not beat that last baseline is neighbourhood structure, not a learned operator.

    flowchart TD
    P["collect n (source, target) pairs of transformation r; record template + entity per pair"] --> S["split by TEMPLATE and by ENTITY family — not random rows (held-out = the transfer test)"]
    S --> F["fit rungs 0-6 on the training split: identity / delta / rank-1 / low-rank / affine / local / MLP"]
    F --> E["on the held-out split, compute the transformation preservation profile per rung"]
    E --> B["baselines: rung 0, random delta, and IGNORE the operator + return the source's nearest neighbour"]
    B --> W{"lowest rung clearing the target bar and the collateral budget, beating all baselines?"}
    W -->|"a rung passes"| SP["simplest passing operator = that rung"]
    W -->|"none passes"| NP["FAIL — no reusable operator for this transformation at this data scale"]
  

Collateral distortion: move the target, spare the rest

An operator that maps its targets perfectly but also shifts every unrelated sentence is not a reusable semantic operator. The knowledge-editing literature learned this the hard way: an edit’s efficacy (did the target change?) has to be weighed against its specificity (did unrelated facts stay put?), and aggressive editing causes measurable “representation shattering” — collateral damage to the surrounding geometry.

So the chapter adds a second requirement, echoing Chapter 21’s preservation profile:

A transformation must move what it is supposed to move without unnecessarily destroying the surrounding geometry.

Measured by applying the fitted operator to a held-out set of items the transformation should not affect, and reporting mean displacement and neighbourhood churn. High collateral drift is not automatically disqualifying — a global “make everything more formal” is supposed to move everything — but it makes the operator a different kind of tool, and that must be recorded, not hidden.

Three levels of evidence: reconstruction → transfer → algebra

reconstruction   the operator fits held-out targets                        weakest
      ↓
transfer         it still works under template / entity / domain shift     stronger
      ↓
algebra          T⁻¹(T(x)) ≈ x  for reversible transformations;            strongest
                 T_a ∘ T_b  matches directly-observed "a and b" examples

An operator that reconstructs but does not transfer is memorization. One that transfers but does not compose may still be operationally useful. A transformation family that transfers, inverts, and composes has genuine geometric structure — this is why RotatE’s headline result is that rotations compose and invert, and why vec2vec trains an explicit cycle-consistency loss (Chapter 18).

Demonstration: the operator bake-off on RELATE transformations

MEASURED on RELATE-DOC v0.1 Family B, Transformation Wave — artifact experiments/embeddings-from-first-principles/wave5/artifacts/operator-bakeoff.json. all-mpnet-base-v2; ladder fitted on paired sentences, held out by base sentence (the content-generalisation test); pass bar = held-out reconstruction cosine ≥ 0.85 and beats the ignore-the-operator baseline.

transformation           simplest passing operator   identity cos   Δ (delta) cos   full-affine / MLP cos
active → passive          IDENTITY                       0.95           0.95            0.39 / 0.53
present → past            IDENTITY                       0.96           0.97            0.31 / 0.48
A acq. B → B acq. A       IDENTITY                       0.98           0.98            0.28 / 0.35
claim → strengthened      IDENTITY                       0.89           0.95            0.48 / 0.55
fact at t1 → t2           IDENTITY                       0.88           0.97            0.44 / 0.54
claim → weakened          CONSTANT DELTA                 0.80           0.94            0.59 / 0.64
formal → informal         NONE PASS                      0.75           0.83            0.57 / 0.62
verbose → concise         NONE PASS                      0.71           0.80            0.35 / 0.45
statement → negation      NONE PASS                      0.67           0.79            0.56 / 0.65

MEASURED. There is no transformation on the list for which a rank-1, low-rank, affine, local, or nonlinear operator does meaningfully better than a constant delta. The expressive operators are strictly worse: a full 768×768 affine map or an MLP fitted on 10–45 sentence pairs lands on the far side of the sphere (reconstruction 0.3–0.6, mean displacement ~1.0). The taxonomy splits cleanly in two:

• Grammatical / role / time / claim-strength edits (passive, past, acquirer↔target, strengthened, temporal shift) are IDENTITY — the transformation barely moves the embedding. There is no operator to learn because there is nothing to transform: “Helios acquired Pine” and “Pine acquired Helios” are cosine 0.98 apart even before any map is fitted.
• Register and length edits (informal, concise) and negation have NO passing operator at all — these are the transformations where the embedding genuinely does move (identity cosine 0.67–0.75), and no operator on the ladder (up to the MLP) captures the move at this data scale.

The delta/analogy tradition assumed a middle case — a consistent, reusable geometric direction. RELATE-DOC v0.1 does not exhibit it for any of the nine transformations. Where a simple operator “works” it is because the edit was invisible; where the edit is real, no simple operator works.

Inverse consistency (row T.4): for the reversible transformations, a fitted ridge map’s T⁻¹(T(x)) returns to cosine 0.20–0.30 of the original — the forward map alone only reaches ~0.35, so round-trips lose most of the signal. Composition and inversion require a genuinely structured operator (RotatE’s rotations), which none of these transformations supplies.

Caveat: RELATE-DOC v0.1 has 10–45 pairs per transformation type, far too few to fit a high-capacity operator. The finding “expressive operators are unfittable here” is real at this scale; whether a mid-complexity operator ever helps negation or concise on hundreds of pairs is a v0.2 question. The finding “grammatical edits are identity” is scale-independent — it is a property of the embedding geometry.

Operators as retrieval keys

For transformations that do have a simple passing operator, an editorial memory can index and retrieve by the operator, not just the text:

new edit request: (source sentence s, desired transformation r)
  candidate past edits = retrieve stored (a, b) pairs where
    the operator fitted to (a, b) is close to the operator for r      (same KIND of edit)
    OR  cos(E(a), E(s)) is high                                       (similar CONTENT)
  → operator-based retrieval surfaces same-kind edits;
    source-based retrieval surfaces similar-content edits;
    they are complementary — use both

The measurable claim is narrow: for relations with a simple passing operator, operator-similarity retrieves same-kind transformations better than source-similarity alone. On RELATE-DOC v0.1 this was not testable — the Transformation Wave found no transformation with a passing operator above the identity/delta rung, so there is no non-trivial operator to key on. The claim survives as a design principle for a model and corpus where a mid-complexity operator does transfer; it is not something this book can now demonstrate.

What this chapter establishes and what it does not

Establishes: the operator ladder T(x) = Wx + b and its rungs; that a delta is W = I and a linear bridge is b = 0 — the alignment and transformation ladders are one ladder; the operator bake-off protocol with the analogy-critique baselines; collateral distortion as a second, independent requirement; the reconstruction → transfer → algebra evidence hierarchy; that the internals literature supplies vocabulary and failure modes but not sentence-embedding evidence.

Does not establish: that any transformation has a reusable mid-complexity geometric operator — the Transformation Wave found none on RELATE-DOC v0.1 (every transformation is identity or has no passing operator; the expressive rungs are unfittable at 10–45 pairs and land on the far side of the sphere). It does establish, measured, that grammatical / role / time / claim-strength edits do not move a sentence embedding (passive, past, acquirer↔target: identity cosine 0.95–0.98), and that register, length, and negation edits do move it but have no operator that transfers (identity cosine 0.67–0.75, nothing on the ladder clears 0.85). Whether a mid-complexity operator ever helps on hundreds of pairs per transformation is a RELATE-DOC v0.2 question. Operator-key retrieval (row T.5) was not run — there is no transformation with a “simple passing operator” other than identity/delta, so the premise does not hold on this corpus.

Lab 23: run the operator bake-off

PROPOSED, not executed.

Setup. 4+ transformation relations, 150+ (source, target) pairs each with template and entity families, one embedding model.

Task.

1. Split by template and by entity (held-out families).
2. Fit rungs 0–6 (identity, delta, rank-1, low-rank r∈{4,16}, full affine, local affine k∈{4,8}, 2-layer MLP) on the training pairs.
3. Per rung, per relation, held-out: target reconstruction cosine; target Top-1 identification; template transfer; entity transfer; collateral drift (on items the transformation should not touch); inverse consistency (reversible relations); composition consistency (compound relations).
4. Report the simplest_passing_operator per relation and its Transformation Complexity label.

Success criterion. A per-relation operator-complexity taxonomy (direction / affine / local / conditioned / non-reusable), each with the margin over the ignore-the-operator baseline and the collateral-drift number. A relation with no rung clearing the bar under the collateral budget is a result, not a failure of the lab.

Companion component: the transformation record

Replaces the delta-centric transformation index:

transformation_record:
  relation:              <relation id>
  space_hash:            <exact space>
  operator:
    class:               <null | delta | rank1 | low_rank(r) | affine | local(k) | nonlinear | conditioned>
    parameter_hash:      <...>
    complexity:          <parameter count / rank / class label>
  training:
    n_train / n_test:    <int / int>
    coverage:            <domains / templates / entities>
    split:               <template | entity | domain>
  preservation:
    target_reconstruction: <...>
    target_identification: <...>
    template_transfer:     <...>
    entity_transfer:       <...>
    domain_ood:            <...>
    collateral_drift:      <displacement + neighbourhood churn on unaffected items>
    inverse_consistency:   <cos(T⁻¹(T(x)), x)>  | n/a
    composition_consistency: <cos(T_a(T_b(x)), observed a∧b target)>  | n/a
    sample_efficiency:     <held-out score vs n_train>
    seed_stability:        <variance across fits>
  simplest_passing_operator: <lowest rung clearing (ε, δ)>
  transformation_complexity: <DELTA | LOW-RANK | AFFINE | LOCAL AFFINE | NONLINEAR | FAIL>
  usable_for:            [ operator-retrieval of same-kind edits, batch restyle, ... ]
  not_usable_for:        [ anything where collateral_drift exceeds the budget, ... ]

The Observatory exposes operator-based transformation retrieval only for relations whose simplest_passing_operator is not FAIL, and records the class.

Failure modes

• Treating the delta as the object. It is rung 1 of a ladder. Fit the ladder.
• Claiming transfer without the ignore-the-operator baseline. “Adding the operator helps” means nothing until it beats “return the nearest neighbour of the source” (Linzen).
• Reporting reconstruction without collateral drift. An operator that also moves everything else is a different tool; say so.
• Random-row splits. Held-out template and entity families are the transfer tests; random rows leak the pattern.
• Importing internals results. A linear “truth direction” in a model’s residual stream is not evidence about a sentence encoder’s geometry.
• Operator computed across spaces. Like a delta, an operator is bound to one space_hash.

What this chapter established

• The operator ladder T(x) = Wx + b — identity, delta, rank-1, low-rank, affine, local, nonlinear, conditioned — and the two connections: a delta is W = I, a linear bridge is b = 0.
• Knowledge-graph models (TransE → TransH → TransR → RotatE) climbed the same ladder for the same reason: complex relations need a matrix, not a vector.
• The operator bake-off, with the analogy-critique baselines (including “ignore the operator, return the source’s neighbour”).
• Collateral distortion as a second, independent requirement: move the target, spare the rest.
• Three levels of evidence: reconstruction → transfer → algebra (inverse and composition consistency).
• The internals literature gives vocabulary and failure modes, not sentence-embedding evidence.
• The transformation_record and the Transformation Complexity label — “the simplest operator that passes.” Measured on RELATE-DOC v0.1: for sentence embeddings that label is identity (grammatical / role / time / claim-strength edits) or none (register, length, negation) — the graded middle of the ladder was not observed for any of the nine transformations at this data scale.

Next

Every part of the book produced an artifact — space records, calibration records, neighborhood reports, bridges, preservation profiles, compression records, transformation records. The final chapter assembles them into one system: an Embedding Observatory that carries the book’s limitations as metadata, and enforces the principle every part of the second half has been circling — every transformation of an embedding creates an obligation to measure what was preserved.