From Vectors to Symbols — The Binding Problem Inside Neural Networks

The previous chapter changed our scale of analysis.

We stopped asking only what architecture a model had and started asking where the computation lived.

Was capability coming from new parameters?

More recurrent steps?

More sampled trajectories?

A selector?

A verifier?

Different post-training?

That wider view exposed a second question.

Suppose the computation works.

Suppose a neural network solves arithmetic, manipulates code, follows grammatical structure, or reasons over logical relations.

What kind of internal representation makes that possible?

A familiar answer is:

Neural networks represent concepts as vectors.

That answer is useful.

It is also incomplete.

A vector for cat is one thing.

A representation for:

cat = subject

dog = object

is another.

The difference is structure.

And structure creates a problem that a simple collection of concept vectors cannot solve.

This chapter begins with that problem, builds one mathematical solution from first principles, and then asks a more surprising question:

Can ordinary neural networks discover this kind of symbolic structure by themselves?

A 2026 paper by R. Thomas McCoy, Paul Soulos, Tal Linzen, and Paul Smolensky provides strong evidence that, in a range of neural networks and symbolic tasks, they can.

The result does not mean that a language model contains a hidden Prolog interpreter.

It does not mean that every neural representation is symbolic.

And it does not prove that Tensor Product Representations are the unique mechanism implemented inside every model.

The result is more interesting than any of those claims.

It suggests that continuous vector systems can develop internal organization that is well described as systematic role–filler structure, and that this structure can be causally involved in model behaviour.

To understand why that matters, we need to start much smaller.

1. A concept vector is not yet a structured representation

Suppose we have learned vector embeddings for three concepts:

cat
chase
dog

Write them as:

cats chase dogs

would be to add the three vectors:

But now consider:

dogs chase cats

Using the same rule gives:

Yet they mean different things.

The problem is not specific to language.

Consider arithmetic:

4 - 7
7 - 4

The same symbols appear.

The result changes because their roles change.

Or code:

concatenate(x, y)

versus:

concatenate(y, x)

Or logic:

if P then Q

versus:

if Q then P

The contents are not enough.

We need to know where each thing belongs.

That is the first pressure that forces us beyond a bag of concepts.

2. The binding problem

Let us name the missing operation.

We need to bind a value to a role.

For the sentence:

cats chase dogs

we want something like:

subject → cats
verb    → chase
object  → dogs

The fillers are:

cats
chase
dogs

The roles are:

subject
verb
object

A representation of the sentence needs to preserve both.

This is the binding problem:

How can a distributed neural representation encode not only which pieces of information are present, but which role each piece occupies?

A good binding mechanism should satisfy several requirements.

It should distinguish:

subject → cat
object  → dog

from:

subject → dog
object  → cat

It should allow us to reuse the same filler in different roles.

cat as subject
cat as object
cat as modifier

It should also allow us to reuse the same role with different fillers.

subject → cat
subject → dog
subject → scientist

And ideally it should generalize to combinations we never explicitly stored.

If we know scientist and we know subject, we should not require a separate indivisible concept named:

SCIENTIST_AS_SUBJECT

for every possible word and position.

That requirement gives us a clue.

The representation should be factorized.

The filler and role should remain reusable pieces.

3. Why atomic role–filler concepts are not enough

We could solve the order problem by inventing a new concept for every combination.

Instead of:

But we have created a new one.

Suppose our vocabulary has 50,000 fillers and 100 possible roles.

A fully atomic scheme may require representations for as many as:

More importantly, the representation no longer tells us how two combinations are related.

The model has no structural reason that:

scientist-as-subject

should be predictable from:

scientist-as-object

and:

doctor-as-subject

The pieces are entangled in the identifier itself.

A more systematic representation would say:

scientist
+
subject
+
a binding operation

Then new combinations can be constructed from known parts.

That is the central idea we will build next.

4. Roles and fillers as vectors

Give every filler a vector.

For a tiny example:

But we still need an operation that says:

cat IN subject role

rather than merely:

cat AND subject

Adding them is not enough.

We need a binding operation.

One mathematically clean option is the tensor product.

5. The tensor product as a binding operation

For two vectors:

subject → cat
object  → dog

Swap the fillers:

The same roles are present.

But the binding changed, so the representation changed.

That is exactly what the additive concept representation could not express.

6. Build it in PyTorch

The smallest implementation is almost embarrassingly simple.

import torch

cat = torch.tensor([1.0, 0.0])
dog = torch.tensor([0.0, 1.0])

subject = torch.tensor([1.0, 0.0])
obj = torch.tensor([0.0, 1.0])

cat_subject = torch.outer(cat, subject)
dog_object = torch.outer(dog, obj)

sentence = cat_subject + dog_object

print(sentence)

Output:

tensor([[1., 0.],
        [0., 1.]])

Now swap the bindings:

dog_subject = torch.outer(dog, subject)
cat_object = torch.outer(cat, obj)

swapped = dog_subject + cat_object

print(swapped)

Output:

tensor([[0., 1.],
        [1., 0.]])

The important point is not the particular numbers.

The important point is the compositional rule:

filler × role
      ↓
   binding

followed by:

binding 1
binding 2
binding 3
    ↓
   sum
    ↓
whole structured representation

This is a Tensor Product Representation, or TPR.

7. The representation can look meaningless and still be structured

Our tiny example is too convenient.

The matrix is easy to read because we chose one-hot-like vectors.

Real neural representations do not usually look like this.

Imagine flattening the matrix into a vector:

[-1.731, 0.082, 4.219, -0.447, ...]

Nothing about those numbers visibly says:

subject → cat
object  → dog

But the underlying geometry can still preserve the same organization.

A rotation does not destroy a square.

A translation does not destroy the relative arrangement of points.

A stretching transformation can make the shape harder for us to recognize while leaving its relations recoverable by another learned transformation.

This gives us an important interpretability rule:

Human unreadability is not evidence that structure is absent.

A network does not need its internal vectors to be legible to us.

It needs later layers to be able to use them.

The linearly transformed TPR used in the paper can be written as:

For each constituent:

1. represent the filler with a vector;
2. represent its role with another vector;
3. bind them with a tensor product;
4. sum the resulting role–filler bindings;
5. transform the combined structure into the model’s representational space.

The result is still a vector.

But it is no longer merely a bag of concepts.

It encodes relations between concepts.

8. Unbinding: can we ask who occupies a role?

A useful structured representation should not only be constructible.

We should also be able to query it.

Suppose:

Conceptually:

whole representation
        +
   subject role
        ↓
      unbind
        ↓
       cat

This matters because the representation is not merely storing a conjunction:

cat exists
dog exists
subject exists
object exists

It supports a question with relational content:

Which filler occupies the subject role?

That is a much stronger representational capability.

We have moved from features to structure.

9. But neural networks were not explicitly built this way

So far we have constructed the representation ourselves.

That proves only that vectors can encode symbolic structure.

The much harder question is whether ordinary neural networks trained with gradient descent actually develop anything like it.

This is where the recent paper becomes interesting.

McCoy and colleagues study networks that were not given an explicit TPR module and ask whether their learned representations can nevertheless be approximated by TPR-like structure.

They study several scales of system.

At the small end are networks trained on synthetic sequence-manipulation tasks.

The architectures include:

MLPs
RNNs
Transformers

At the larger end they analyze seven language models, including Gemma 3, GPT-2-XL, GPT-OSS-20B, Pythia-12B, Qwen3-14B, OLMo-2-13B, and Llama 3.1-8B.

They also study GPT-OSS performing tasks from domains traditionally treated as symbolic:

arithmetic
logic
computer code
language

The paper’s core hypothesis is not merely that information about roles can be decoded.

It is that the network representations themselves can be closely approximated by systematic role–filler structure.

To test that, the authors use a method called DISCOVER:

DISsecting COmpositionality in VEctor Representations

10. DISCOVER asks a stronger question than a probe

A linear probe usually asks something like:

Can I train a classifier to recover property X from this hidden state?

Suppose a probe can detect whether a sentence is plural.

That tells us plural information is available to the probe.

But it leaves an important ambiguity.

The information may be present without being the representation the original model actually relies on.

DISCOVER uses a more demanding test.

Imagine a target model with:

input
  ↓
encoder
  ↓
hidden representation E
  ↓
decoder
  ↓
output

Now build an explicit TPR-based approximation:

input structure
      ↓
roles + fillers
      ↓
Σ filler ⊗ role
      ↓
W(...)+b
      ↓
TPR approximation E_TPR

Train the TPR approximation so that:

Throw away the target model’s original encoder output.

Feed the explicit TPR approximation into the original decoder:

input
  ↓
explicit TPR equation
  ↓
E_TPR
  ↓
original decoder
  ↓
output

If the decoder still produces the correct result, then the TPR is not merely correlated with some property of the hidden state.

It is acting as a functional replacement for that hidden state.

That is a substantially stronger result.

11. A toy version: reverse a sequence

The paper uses a sequence-reversal model as a running example.

Imagine a network trained on:

Q M Z → Z M Q

The encoder must somehow represent:

Q is first
M is second
Z is third

because the decoder cannot reverse the sequence correctly if it knows only that the three letters occurred.

A role scheme might be:

1st → Q
2nd → M
3rd → Z

Then the explicit representation becomes:

Then the original decoder receives E_TPR instead of the encoder’s own hidden state.

The question is simple:

Can the original network still do its job?

Across the model classes studied, the answer is often yes with high accuracy.

That suggests the TPR equation has captured something functionally important about the learned representation.

12. Why representation replacement matters

There is a hierarchy of interpretability evidence.

At the weak end:

I found a correlation.

Stronger:

I can decode a feature.

Stronger again:

I can reconstruct the hidden representation.

More compelling:

I can replace the original representation
with my structured approximation
and downstream behaviour survives.

And stronger still:

I can edit one structured component
and make downstream behaviour change
in the predicted way.

The paper climbs this ladder.

That is why it is more interesting than a result that merely says:

We found vectors correlated with syntax.

The key issue is causal use.

13. Constituent surgery: edit the representation, change the behaviour

Consider the sentence:

The clever doctor helped the lawyer.

A structured representation might include:

doctor  → subject noun
clever  → subject adjective
lawyer  → object noun

Now change only one binding.

Move clever from:

subject adjective

to:

object adjective

The resulting structure corresponds to:

The doctor helped the clever lawyer.

Because TPRs are additive over role–filler bindings, a local edit can be written conceptually as:

If the network behaves as though the adjective has moved, we have much stronger evidence that the discovered structure participates in the computation.

This is what the authors test with targeted causal interventions.

Across six GPT-OSS task settings, they report 31 types of interventions with an average intervention accuracy of 0.903.

That number is striking for two reasons.

First:

0.903 is high enough

that the symbolic decomposition is clearly doing substantial explanatory work.

Second:

0.903 is not 1.000

which warns us not to collapse the network into an exact symbolic machine.

We will return to that distinction.

14. Filler edits make the causal test concrete

The same idea can be used in arithmetic.

Suppose the model is working with:

-2 + 3 * -4

The representation contains some binding corresponding to the filler 3 in a particular operand role.

Change that filler to 8 while leaving the surrounding structure intact.

The represented expression becomes:

-2 + 8 * -4

The expected answer changes from:

-14

to:

-34

The paper reports high accuracy for these kinds of targeted filler interventions.

This matters because the intervention is not merely changing the entire hidden state arbitrarily.

It is changing one hypothesized constituent while preserving the rest of the structure.

That is the kind of experiment we want from mechanistic interpretability:

hypothesis about mechanism
        ↓
controlled internal edit
        ↓
predicted behavioural consequence
        ↓
observe whether prediction holds

15. The decisive test is novel role–filler combinations

There is still a loophole.

A TPR-shaped model could cheat.

Imagine that DISCOVER simply learns an independent representation for every observed pair:

cat-as-subject
dog-as-object
scientist-as-object

Then it might fit the training representations without learning genuinely factorized roles and fillers.

To test for systematicity, we need to withhold combinations.

Suppose DISCOVER is allowed to see:

scientist → object
scientist → modifier

doctor → subject
lawyer → subject

but never:

scientist → subject

Now test it on exactly that unseen pairing.

If scientist and subject are genuinely reusable factors, the representation should still be constructible.

If every pair is an atomic memorized unit, there is no basis for constructing the unseen combination.

This is the key generalization test.

The paper reports robust above-baseline generalization to unseen role–filler combinations across almost all of the target-model settings studied, with arithmetic being an important weaker case in the GPT-OSS analysis.

That last caveat matters.

The result is not:

all networks always develop perfect compositional structure

It is:

systematic role–filler structure appears broadly enough
that an explicit compositional approximation can generalize
beyond the combinations on which it was fitted

That is a much more defensible statement.

16. Representation compositionality is not the same as behavioural generalization

At first this may seem contradictory.

Neural networks are famous for sometimes failing at compositional generalization.

If their representations contain systematic role–filler structure, why do they still fail when familiar elements occur in unfamiliar combinations?

Because two claims are different:

Or the representation may be only approximately systematic.

Or different parts of the network may use different overlapping role schemes.

Or training may produce a representation that supports some novel combinations much better than others.

This distinction is extremely useful.

It prevents us from treating one successful interpretability result as a universal behavioural theorem.

A model can possess structured ingredients without possessing a perfectly general symbolic algorithm.

17. Do not conclude that the network literally contains our equation

We now need to apply the same discipline we used throughout this book.

An interpretable approximation is not automatically an ontological identity.

If:

That is stronger than saying a probe can decode something.

But it still does not prove that the network contains a literal module whose source code is:

for filler, role in structure:
    state += torch.outer(filler, role)

The network may implement an equivalent or closely related geometry through distributed computation across attention, MLPs, residual streams, and other mechanisms.

Several representational formalisms can share important mathematical properties.

So the safe claim is:

TPR-like role–filler structure is a useful and causally supported description of the learned representation.

The unsafe claim is:

We found the exact hidden symbolic program.

Those are not the same statement.

18. Approximately symbolic may be more accurate than symbolic

The intervention accuracy gives us another reason for restraint.

If the model were an exact symbolic system operating exactly on the discovered structure, we might expect perfect intervention behaviour under the experiment’s assumptions.

Instead the average reported GPT-OSS intervention accuracy is:

The model may be neither:

purely unstructured statistical soup

nor:

an exact discrete symbolic computer

A better picture may be:

continuous neural state
        ↓
strong systematic geometry
        ↓
approximately symbolic organization
        ↓
behaviour that is structured but not perfectly discrete

The authors connect this interpretation to a view they call limitivism: neural systems may approach symbolic structure without becoming exact classical symbolic systems.

For us, the important engineering lesson is simpler.

Do not force a binary choice between “neural” and “symbolic.”

A continuous system can implement organization that behaves symbolically enough to support structured computation while remaining approximate, distributed, and learned.

19. This changes what “distributed representation” means

The phrase distributed representation is sometimes interpreted too loosely.

It can sound like:

Everything is smeared everywhere, so there is no meaningful structure.

That conclusion does not follow.

A representation can be distributed and structured.

Consider a map.

Rotate it 37 degrees.

Stretch it horizontally.

Translate it upward.

The coordinate values of every point change.

The map may become awkward for a human to inspect numerically.

But distances, alignments, and other recoverable relationships can remain systematic.

The same principle applies in higher-dimensional neural spaces.

A vector with thousands of unfamiliar floating-point values can still participate in a highly organized geometry.

The right question is not:

Can I read meaning directly from dimension 847?

The better question is:

What transformations, subspaces, relations, and compositional operations remain stable enough for the network to use?

That is a much richer view of representation.

20. From features to relations

Earlier interpretability work often asks whether a hidden vector contains a feature such as:

plural
French
Golden Gate Bridge
Python code
positive sentiment

Those questions are useful.

But intelligent computation frequently depends on something more structured:

A is larger than B
X is the argument of function F
7 is the denominator
cat is the subject
P is the antecedent
Q is the consequent

These are not merely features.

They are relations.

A feature-based picture asks:

What concepts are present?

A structured picture also asks:

How are those concepts arranged?

That is the conceptual advance of this chapter.

We began with vectors as containers of information.

We now need vectors as containers of organized relationships.

21. What gradient descent may be discovering

Nobody explicitly programmed the analyzed language models with:

subject role vector
object role vector
numerator role vector
function argument role vector

The models were trained by optimizing objectives over data.

Yet a structured approximation can recover reusable roles and fillers from their hidden states.

This suggests a powerful possibility:

Gradient descent can discover representational organizations that solve recurring structural pressures in the data.

Language repeatedly asks:

who did what to whom?

Arithmetic repeatedly asks:

which value occupies which operand position?

Code repeatedly asks:

which expression is the function?
which expression is the argument?
which variable is bound where?

Logic repeatedly asks:

which proposition is premise?
which is consequence?

A role–filler representation is useful across all of these because the same generic pressure appears repeatedly:

identity alone is insufficient
position / relation matters

We should not be surprised if optimization discovers reusable structure when reusable structure reduces the difficulty of the task.

What is surprising is that we are beginning to obtain methods capable of exposing that structure after training.

22. A small experiment we can reproduce ourselves

We do not need a 20-billion-parameter language model to understand the idea experimentally.

We can build a small sequence task.

Task

Generate sequences such as:

A D C
B A E
C E B

Train a model to reverse them:

A D C → C D A
B A E → E A B
C E B → B E C

The model must preserve both:

which letters occurred

and:

where each letter occurred

Model

Use a tiny encoder-decoder network:

input tokens
    ↓
embedding
    ↓
encoder
    ↓
latent vector E
    ↓
decoder
    ↓
reversed sequence

Analysis

After training, freeze the model.

Create a TPR approximation:

f_token = learned filler embedding
r_i     = learned position-role embedding

Train only the TPR approximation to minimize:

Perform the functional test:

E_TPR
  ↓
original decoder
  ↓
reversed output

Measure exact sequence accuracy.

This gives us our own miniature version of the paper’s central experiment.

23. Then make the experiment harder

Once the basic approximation works, withhold role–filler combinations.

Suppose sequence length is three.

During training of the TPR approximation, never show:

A in position 1
C in position 2
E in position 3

But allow the filler vectors to see those letters elsewhere and the role vectors to see those positions with other letters.

Then test:

A B D
B C A
D A E

Now we are no longer asking whether the approximation can memorize combinations.

We are asking whether it can compose known fillers with known roles in unseen pairings.

That is the experiment that distinguishes:

memorized pair inventory

from:

systematic binding

A strong chapter notebook should report both.

24. Add causal surgery to the notebook

We can go one step further.

Take a sequence:

A D C

Its TPR approximation contains:

position 1 → A
position 2 → D
position 3 → C

Now edit the first filler from A to B:

position 1 → B
position 2 → D
position 3 → C

The edited represented input is:

B D C

The original decoder should now produce:

C D B

This gives us a tiny causal intervention experiment.

If it succeeds, we have shown three things in one notebook:

1. structured approximation
2. generalization to unseen bindings
3. causal manipulation of a constituent

That is much closer to genuine mechanistic evidence than visualizing a few embeddings and declaring them meaningful.

25. The experiment needs controls

A first-principles experiment should make failure interpretable.

So we need baselines.

Baseline 1: atomic role–filler embeddings

Give every observed pair its own embedding:

A_at_position_1
A_at_position_2
B_at_position_1
...

This model can fit seen combinations extremely well.

But when a withheld pair appears, it has no reusable composition rule.

That makes it a useful contrast.

Baseline 2: filler-only representation

Ignore position:

Baseline 3: shuffled roles

Preserve fillers but randomly permute role assignments.

If performance remains unchanged, our proposed roles are probably not doing meaningful work.

Baseline 4: original latent state

Always retain the decoder’s performance using the untouched encoder representation.

That gives us the ceiling we are trying to approximate.

The useful table becomes:

The question marks are important.

We should run the experiment rather than write the desired result into the chapter.

26. What would falsify our interpretation?

Suppose the TPR approximation reconstructs hidden vectors well but the original decoder performs poorly when given those approximations.

Then geometric similarity is not enough.

Suppose decoder performance is high on seen role–filler pairs but collapses on unseen combinations.

Then DISCOVER may have fitted atomic pair structure rather than systematic binding.

Suppose causal edits change behaviour, but the direction of change is unrelated to the edited constituent.

Then the intervention is perturbing the network without validating our semantic interpretation.

Suppose shuffled roles perform just as well as the hypothesized roles.

Then our chosen role scheme has not earned its explanatory status.

These are not nuisances.

They are the experiment.

A useful interpretability hypothesis should expose ways it can fail.

27. This is not yet the mechanism that creates the structure

Even if we establish that a hidden state has TPR-like organization, a major question remains unanswered.

How did the network construct it?

In a Transformer, the representation may have been produced by many layers of:

attention
residual addition
normalization
MLP transformations

The TPR description tells us something about the state that exists.

It does not yet tell us the complete circuit that generated that state.

That distinction mirrors one we made earlier in the book.

A score tells us what a model outputs.

It does not automatically tell us how the score was computed.

Likewise:

structured representation
        ≠
complete generating mechanism

This gives mechanistic interpretability its next task.

If role–filler structure exists, which attention heads, MLPs, residual directions, or recurrent updates build it?

Which components read it?

Where is it first formed?

Where is it transformed?

Where does it disappear?

Those are different questions from whether the representation can be described structurally.

28. Representation and computation form two layers of explanation

We can now separate two kinds of explanation.

Representational explanation

What information and relationships are encoded in the hidden state?

subject → cat
object  → dog

Computational explanation

What operations caused that representation to appear and how is it subsequently used?

attention head A
      +
MLP feature transformation
      +
residual update
      ↓
role–filler geometry
      ↓
later attention / MLP reads it

A complete model explanation eventually needs both.

But keeping them separate is useful because each can be tested differently.

DISCOVER-like methods attack the representational question.

Circuit-level interventions attack the computational question.

Together they may eventually give us something much closer to a source-level explanation of learned systems.

29. The program may exist as geometry

At the beginning of this book we repeatedly opened model abstractions.

A model became submodels.

Submodels became blocks.

Blocks became tensor operations.

Eventually the impressive name disappeared and only explicit computation remained.

This chapter adds another direction of decomposition.

Instead of opening the architecture, we open the representation.

The naive extremes are:

There is no program.
It is all statistics.

and:

There must be an ordinary symbolic program
hidden somewhere inside the weights.

The evidence we have examined suggests a third possibility.

The model may implement structured operations through geometry.

Not Python AST nodes.

Not explicit logical predicates stored as discrete objects.

Not unstructured floating-point noise either.

But reusable entities and relationships encoded in continuous space and transformed by learned computation.

In that sense:

The program may exist as geometry.

That is not a mystical claim.

It is a concrete hypothesis about representational organization that can be tested by reconstruction, substitution, generalization, and intervention.

30. What this paper establishes — and what it does not

Let us separate the claims carefully.

Strong evidence from the paper

Across multiple neural architectures and language models, the authors find representations that can be closely approximated by linearly transformed TPRs.

They show that these approximations can often replace original learned representations while preserving downstream behaviour.

They show targeted causal interventions in which changing parts of the discovered structure changes model behaviour in expected ways.

They test unseen role–filler combinations and find substantial evidence for systematic binding rather than only atomic pair memorization.

What remains open

The result does not show that every representation in every neural network is TPR-like.

It does not show perfect symbolic behaviour.

It does not show that the TPR equation is the unique internal implementation.

It does not fully explain which Transformer circuits create and consume the structure.

It does not eliminate known failures of compositional generalization.

And it does not turn interpretability into a solved problem.

The responsible conclusion is narrower and more valuable:

Continuous neural representations can develop systematic, reusable role–filler structure that is both functionally useful and causally implicated in behaviour.

That is enough to change how we think about the relationship between vectors and symbols.

31. The architecture ladder now gains a representation axis

Until now, our book’s decomposition looked roughly like this:

model
  ↓
submodels
  ↓
blocks
  ↓
layers
  ↓
tensor operations

Now add another axis:

hidden state
    ↓
features
    ↓
roles + fillers
    ↓
bindings
    ↓
structured geometry

These two axes meet.

architecture
    ↓
computation
    ↓
hidden state
    ↓
representation
    ↓
behaviour

A model becomes understandable only when we can connect those levels.

Architecture alone is not enough.

Representation alone is not enough.

Behaviour alone is not enough.

The mechanism lives in the chain between them.

32. What we learned

We began with a simple limitation.

Adding concept vectors cannot distinguish:

cats chase dogs

from:

dogs chase cats

because the concepts are the same while the bindings differ.

That forced us to introduce roles and fillers.

We then used the tensor product to bind reusable fillers to reusable roles:

DISCOVER asks whether a learned neural representation can be replaced by an explicit TPR-like equation while preserving downstream behaviour.

Causal interventions ask something stronger: if we edit one constituent in the discovered structure, does the model’s behaviour change as predicted?

Generalization to withheld role–filler combinations asks whether the decomposition is genuinely systematic rather than an inventory of memorized atomic pairs.

Together these tests support a richer picture of neural representations.

They can be:

continuous
+
distributed
+
high-dimensional
+
approximately symbolic
+
systematically compositional

Those properties are not mutually exclusive.

The old opposition:

symbols OR vectors

is too simple.

A vector space can carry symbolic organization.

A neural network can learn that organization without an explicit symbolic module.

And an interpretable symbolic approximation can sometimes let us manipulate the learned system in controlled ways.

That brings us to the next question.

We now have evidence about what kind of structure may exist inside a hidden state.

But we still need to know:

Which neural computations build that structure, move it through the network, and use it to produce the next result?

That is where representation analysis becomes circuit analysis.

References

• R. Thomas McCoy, Paul Soulos, Tal Linzen, and Paul Smolensky. The Emergent Symbolic Structure of Artificial Neural Networks. arXiv:2608.29530, 2026. https://arxiv.org/abs/2608.29530
• Paul Smolensky. Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems. Artificial Intelligence, 1990.
• R. Thomas McCoy et al. DISCOVER work on compositional structure in neural representations, extended in the 2026 paper above.