Semantic Enhancement

A team inherits a ten-year-old payments service and its forty-page architecture document. One passage describes the retry policy in dense prose: three paragraphs of conditions, backoff intervals, idempotency keys, and two exceptions buried in the middle. A new engineer reads it three times and still cannot say what happens when the third retry fails during a partition. Then someone draws seven boxes and nine arrows on a whiteboard (states, transitions, the two exception paths in red) and the same engineer answers correctly in thirty seconds. Same passage. Same reader. Radically different understanding.

Two pages later the same trick fails. A different passage compares four database options across eleven dimensions, with exact throughput figures, consistency guarantees and cost bands. Someone renders it as an attractive isometric illustration, database cylinders floating over clouds. It is beautiful and useless. The numbers that decide the choice are gone, and the comparison the prose made laborious is now impossible rather than merely difficult. The right enhancement for that passage was a table, or nothing at all, since the table it already contained worked fine.

The last chapter asked how a smaller representation could help a reader decide whether to enter a source. This one asks about the reader who is already inside it. The page stays in front of them, and the representation changes locally to make something easier to see. When does that help, and what must it not do?

FIXED PAGE
    ↓
PREVIEW: compress the source before entry
    ↓
ENHANCEMENT: modify the representation during use

Three things an enhancement can do

The word “enhancement” covers three different acts, and the chapter’s central distinction is among them. Take the retry passage.

Reveal. Make structure that is already in the source perceptually obvious. Highlight the two exceptions. Bold the condition that triggers each backoff. Number the steps. Nothing is said that the passage did not say; the reader just finds it faster.

Re-express. Render information the source already contains in a different form. The state diagram of the retry policy, the table of backoff intervals, a timeline of what happens in each retry window. The form is new, but every claim in it should trace back to the prose.

Add. Introduce information that is not in the source. A definition of idempotency for a reader who has never met the word. A link to the paper where the backoff scheme came from. A note that the policy contradicts a later design document.

enhancement  ≠  additional claim

Revealing and re-expressing are forms of mediation: they change how supported information is presented. Adding is different in kind. It brings in a new claim, and a new claim creates an obligation to say where it came from and whether it is any good. Adding can be very valuable, and later chapters build machinery for it. But it is not the same act as making existing structure visible, and a system that blurs them will smuggle unsupported content into a page under the label of “clarification”.

The line is thinner than it looks, because re-expression can slip into addition without anyone noticing. A diagram forces choices the prose did not: which box sits above which, whether an arrow points left or right, whether two items are connected at all. If the arrow runs from the retry to the dead-letter queue and the prose never said so, the diagram has made a claim. This is why re-expression needs checking against the source, claim by claim and relation by relation, and not just a glance to see whether it looks right.

What structure makes cheap

How should a system choose between a table, a diagram, a timeline and plain prose? Not by a general feeling that images help. The chapter’s mechanism is a quiet idea: different representations make different relationships cheap to perceive. A table exposes comparison, with values aligned for scanning and the best row findable in seconds. A timeline exposes sequence and duration: what came first, what overlapped, how long the gap lasted. A diagram exposes topology: states and transitions, containment, direction of dependence, cycles. Prose, unfashionable and undefeated, exposes qualification, argument and nuance: the exception, the scope condition, the “except when”, the reason to believe. An interactive form exposes contingency, what happens if, explored at the reader’s own pace.

This is an old observation in the study of diagrams. Larkin and Simon’s classic paper argued that a diagram can be better than a set of sentences when it makes the needed inferences easy to read off, by keeping related information together and by showing relations directly. The claim worth keeping is a modest one. Under some tasks, making a relationship explicit in the representation can reduce the work needed to recover it. It says nothing about comprehension in general, retention or transfer. Each of those would need its own evidence.

So the question for any passage is never “which medium is best?” but “which relationships does this representation make cheap to perceive, and are those the relationships this passage is about?” The defaults that follow are starting points to be tested, not laws:

a process with topology      → perhaps a diagram
a comparison across options  → perhaps a table
a chronology with intervals  → perhaps a timeline
a hierarchy                  → perhaps a tree
a spatial arrangement        → perhaps a visual
a qualified argument         → probably leave it alone

Note the asymmetry. Five kinds of structure suggest a candidate form, while the sixth, argument with qualification, defaults to leaving the prose as it is. That is deliberate. Prose is the highest-bandwidth channel for hedged, conditional or contested content, and every transformation of such content risks dropping a qualifier, smoothing away uncertainty or upgrading a correlation to a cause. The prior should be conservative, and leaving the passage unchanged should win a large share of the time.

The multimedia principles from earlier now work as engineering constraints. A diagram detached from the prose it explains recreates the split-attention cost that spatial contiguity warns about. An enhancement that repeats the prose point for point pays the redundancy penalty and buys nothing. Decorative visuals that add no structural information violate coherence. And signalling cuts both ways: arrows and emphasis that reveal the passage’s real organisation help, while highlights that merely decorate misdirect attention.

Choosing, and the right to decline

A system that enhances has to make a decision for each passage: what is this passage doing, which form might help, what would that cost and risk, and is it worth it? In outline:

source passage
    ↓
what is it doing? (process, comparison, chronology, hierarchy, argument)
    ↓
candidate forms
    ↓
expected benefit against cost and distortion risk
    ↓
reveal · re-express · add · or leave it as it is

Two separations keep this honest.

Choosing the form and executing it are different skills. A system can pick the right kind of representation and draw a bad one: mislabelled arrows, a missing exception path. That does not show diagrams were the wrong choice. Conversely, a beautiful diagram can hide the fact that leaving the passage alone would have been better, because polish is not fit. A fair assessment scores the choice and the instance separately, so a weak generator does not get diagrams abandoned and a slick one does not get decoration enshrined.

Fit is not preference. The discipline from earlier chapters carries over: comprehension, time, effort and liking are measured apart, and an enhanced page that people like but understand less is a failure.

The hardest output of such a system, and the most praiseworthy, is a decision to leave the passage unchanged. When that is right, it is a success, not a failure to generate something.

Generation is weaker than it looks

One piece of evidence disciplines the whole enterprise, and it concerns the executing half. DiagramEval (EMNLP 2025) evaluates diagrams generated by language models by treating each as a graph: text elements are nodes, connections are directed edges. It scores node alignment (are the right elements there?) separately from path alignment (are the relationships between them right?). The findings are sobering in exactly the dimension that matters here. Current models often produce diagrams whose elements line up with the source while their relationships do not, and path alignment barely correlates with CLIPScore, a standard image-text metric that is sensitive to layout and styling and can rate a structurally wrong diagram well. Put plainly, current generators are comparatively good at drawing the boxes and comparatively poor at drawing the arrows, and the arrows are what a diagram is for.

For a system choosing among forms, the lesson is to discount a diagram’s expected benefit by its generation risk. A table of exact values, whose cells can be checked, carries less of that risk. For anyone judging such systems, the lesson is to measure relations separately from elements, because the field’s default metric blurs the two.

When enhancement helps and when it hurts

Four kinds of passage pull a system in different directions, including one where doing nothing is right. They also cover the failures worth watching for.

1. Prose that is already sufficient. A clear, well-argued passage with its reasoning explicit. Any enhancement adds load without gain. Leaving it alone is correct.
2. An important exception inside prose. A conclusion that holds except under a stated condition: a contraindication, a scope limit, a version-specific caveat. Diagrams tend to delete the qualification. Leaving the passage alone, or a light reveal such as marking the exception, serves better.
3. A comparison with many exact values. Options differing across many precise figures. A table beats an illustration, because comparison wants alignment and exact values must survive.
4. A process with real topology. States, transitions, cycles, exception paths: the payments retry policy. Here a diagram can earn its place, and this is the case that shows whether enhancement ever pays at all.

Across these, the ways an enhancement goes wrong are few and recognisable: a poorly drawn diagram, a relation drawn the wrong way round, annotation so heavy it buries the text, an extra form that costs the reader more attention than it saves, and an inference the source never made, presented as if it had. A system that diagrams everything fails differently from one that timidly changes nothing even for the retry policy. Both are errors, and they are not the same error.

Responding to the task, not building a model of the person

One boundary needs stating now, because it will matter a great deal later. Everything in this chapter responds to the passage in front of the reader and to what they are trying to do with it right now. Nothing here builds a lasting picture of the reader. The enhancement is task-adaptive: it depends on the structure of the material and the local need. It is not personalised, in the sense of depending on a stored model of who the person is, what they know or what they prefer. Those are separate steps with separate risks, and the book takes them in order.

task-adaptive representation  ≠  personalised representation

Where this leaves us

Enhancement can help a reader inside a source, but only by one of three acts, and only two of them are plain mediation. Revealing makes existing structure visible, re-expressing renders existing information in a new form, and adding brings in something new that the source did not say. The first two must be checked against what the source claims; the third needs support of its own. Which form to use depends on which relationships a passage is about and which a representation makes cheap to see, and the best answer is sometimes to leave the passage alone.

Notice what the system has had to do throughout. For every passage it re-derived, from the raw prose, what kind of structure it contained: a process here, a comparison there, an exception buried in the middle. It does this each time from scratch, nothing it learns is kept, and two renderings of the same passage made on different days can quietly disagree. The payments policy might become a diagram on Monday that shows an exception and a table on Friday that omits it. There is nothing for the views to be checked against. That gap, a stable object that sits between the source and every view of it, is the next problem.

References

• Larkin, J.H. & Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65–100.
• Liang, C. & You, J. (2025). Evaluating LLM-Generated Diagrams as Graphs (DiagramEval). Proceedings of EMNLP 2025, 2025.emnlp-main.640. Preprint titled “DiagramEval: Evaluating LLM-Generated Diagrams via Graphs”, arXiv:2510.25761.
• Mayer, R.E. (2021). Multimedia Learning (3rd ed.). Cambridge: Cambridge University Press. (Source of the coherence, redundancy, contiguity and signalling principles used here as engineering constraints.)