Semantic Enhancement

A team inherits a ten-year-old payments service and its forty-page architecture document. One passage describes the retry policy in dense prose: three paragraphs of conditions, backoff intervals, idempotency keys, and two exceptions buried in the middle. A new engineer reads it three times and still cannot say what happens when the third retry fails during a partition. Then someone draws seven boxes and nine arrows on a whiteboard — states, transitions, the two exception paths in red — and the same engineer answers correctly in thirty seconds. Same source passage. Same reader. Radically different understanding.

Two pages later, the same trick fails. A different passage compares four database options across eleven dimensions with exact throughput figures, consistency guarantees, and cost bands. Someone renders it as an attractive isometric illustration with database cylinders floating over clouds. It is beautiful and useless: the numbers that decide the choice are gone, the comparison the prose made laborious is now impossible rather than merely difficult. The right enhancement for that passage was a table — or nothing at all, since the prose table it already contained worked fine.

This chapter is about the difference between those two outcomes. Chapter 4 asked whether a transformed representation helps a reader choose a source; this chapter asks whether an added representation helps a reader understand one they are already consuming. The mechanism is a representation router, and the chapter’s central discipline is that the router’s hardest and most praiseworthy output is UNCHANGED.

The router

The conceptual shape is a pipeline, stated here in full because later chapters reuse its stages:

source passage
    ↓
identify communicative structure
    ↓
candidate representation
    ↓
estimate expected benefit / risk
    ↓
TEXT · IMAGE · DIAGRAM · TABLE · TIMELINE · INTERACTIVE · UNCHANGED
    ↓
measure understanding

Each stage matters. Identify communicative structure asks what the passage is doing: describing a process, comparing options, narrating a chronology, laying out a hierarchy, situating things in space, advancing a qualified argument. Candidate representation proposes forms suited to that structure. Estimate benefit and risk weighs the expected comprehension gain against production cost, attention cost, and — heaviest — distortion risk: what the new form might drop, flip, or smooth. Measure understanding closes the loop with comprehension and transfer tests, never preference alone.

Two separations keep the router honest, and EXP-05 below enforces both.

Routing versus generation quality. Choosing the right representation type and producing a good instance of it are different skills that fail independently. A terrible Mermaid diagram — mislabelled arrows, missing exception path — does not prove diagrams were the wrong choice for a process passage; it proves the generator failed at an instance of the right choice. Conversely, a beautiful diagram can hide the fact that UNCHANGED was the better routing decision: polish is not fit. The experiment therefore scores the two errors separately, with human judges rating routing appropriateness (given the passage structure, was this type defensible?) apart from instance quality (given the type, was this instance faithful and legible?). A book that confuses the two will either abandon diagrams because its generator is weak or enshrine decoration because its generator is slick. Neither mistake survives this separation.

Fit versus preference. Chapter 4’s discipline carries forward unchanged: comprehension, transfer, time, perceived effort, and preference are measured separately, and an enhanced page that is preferred but transfers worse is a meaningful failure. The router optimises understanding per unit of reader effort, not delight.

Structure is the reason: representation affordance

The router must never reason “images tend to help.” It reasons from structure to fit, and the mechanism underneath is a quiet idea this chapter introduces without building a cathedral around it: representation affordance — different representations make different relationships cheap to perceive.

A table exposes comparison: rows against columns, values aligned for scanning, the best row findable in seconds. A timeline exposes sequence and duration: what preceded what, what overlapped, how long the gap lasted. A diagram exposes topology: states and transitions, containment, direction of dependence, cycles. Prose — unfashionable, undefeated — exposes qualification, argument, and nuance exceptionally well: the exception, the scope condition, the “except when,” the reason to believe. An image exposes appearance and spatial composition: what the thing looks like, how parts sit together. An interactive exposes contingency: what happens if, explored at the reader’s own pace.

The question is therefore never “which medium is best” but “which relationships does this representation make cheap to perceive?” — asked about a specific passage whose communicative structure has been identified. Affordance so defined is a perceptual claim only: it predicts what structure becomes cheap to inspect, not what gets comprehended, retained, or transferred — each of those needs its own measures, and EXP-05’s comprehension/transfer instruments test task-bounded understanding, never durable learning. The routing heuristics follow directly, as defaults to be tested rather than laws to be obeyed:

process with topology      → candidate diagram
comparison across options  → candidate table
chronology with intervals  → candidate timeline
hierarchy                  → candidate tree
spatial arrangement        → candidate visual
discursive/qualified prose → probably UNCHANGED

Note the asymmetry: five structures nominate candidates, while the sixth — argument with qualification — defaults to leaving the prose alone. That asymmetry is deliberate. Prose is the highest-bandwidth channel for hedged, conditional, contested content, and every transformation of such content risks precisely the distortion Chapter 9 will price: dropped qualifiers, smoothed uncertainty, upgraded causation. The router’s prior should be conservative, and UNCHANGED should win a large share of encounters. When UNCHANGED wins, that is a successful routing decision, not a failed generation attempt — the scoring says so explicitly.

Mayer’s principles return here in their proper role: engineering constraints, not literature. A diagram whose elements are disconnected from their explanatory prose recreates the split-attention cost spatial contiguity warns against — the reader burns capacity integrating what the enhancement was meant to unify. An enhancement that duplicates the prose point-for-point pays the redundancy penalty for zero gain. Decorative visuals that add no structural information violate coherence: extraneous material that taxes capacity without serving the objective. And signalling cuts both ways — arrows, emphasis, and layout that highlight the passage’s real organisation help; highlights that merely decorate misdirect selection itself. Each constraint is a sentence, applied, and the chapter moves on. Chapter 2 earned them; this chapter spends them.

Generation is weaker than it looks

One more piece of evidence disciplines the router’s ambitions, and it concerns the generation stage specifically. DiagramEval (EMNLP 2025, 2025.emnlp-main.640; code at github.com/ulab-uiuc/diagram-eval) treats LLM-generated diagrams as graphs — text elements as nodes, connections as directed edges — and scores node alignment (are the right elements present?) separately from path alignment (are the relationships between them correct?). The findings are sobering in exactly the dimension this chapter cares about: state-of-the-art models (Llama 4 Maverick, Gemini 2.5 Pro, Claude 3.7 Sonnet in the reported evaluation) produce diagrams whose elements often align while their relationships do not — path-alignment scores show minimal correlation with the standard CLIPScore metric, which is over-sensitive to layout and styling and can rate a structurally wrong diagram well. In other words, current generators are comparatively good at drawing boxes and comparatively bad at drawing the arrows between them — and the arrows are the topology, the thing diagrams are for.

For the router, the lesson is quantitative humility: candidate DIAGRAM carries a generation-risk discount that candidate TABLE (exact values, checkable cells) does not, and the benefit/risk estimate must price it. For the book, the lesson is architectural: routing correctness and instance fidelity need separate instruments, because the field’s default metric conflates them. Chapter 9 will build the fidelity side; this chapter builds the routing side.

Four failure classes the router must discriminate

EXP-05 is designed around passage types that pull the router in different directions — including cases where the right answer is to do nothing. Four classes, each with its predicted winner:

1. Prose already sufficient. A clear discursive passage — a well-structured argument with its reasoning explicit. Prediction: UNCHANGED wins; any enhancement adds load without gain. This class calibrates the router’s restraint.
2. Important exception embedded in prose. A passage whose conclusion holds except for a stated condition (a contraindication, a scope limit, a version-specific caveat). Prediction: diagrams risk deleting the qualification; UNCHANGED or minimal TEXT reorganisation wins. This class tests distortion awareness.
3. Comparison with many exact values. Options differing across numerous precise figures. Prediction: TABLE beats IMAGE and DIAGRAM; prose alone loses on scanning time. This class tests affordance matching — comparison wants alignment, not illustration.
4. Process with important topology. States, transitions, cycles, exception paths — the payments retry policy. Prediction: DIAGRAM genuinely beats prose on comprehension and transfer per unit time. This class tests whether enhancement ever earns its keep at all.

A router that cannot separate these four — that diagrams everything, or that timidly changes nothing even for the retry policy — has failed differently in each direction, and the scoring distinguishes cowardice from recklessness. Both are errors; they are not the same error.

Keeping Chapter 6 hungry

A final boundary, structural rather than evidential. Notice what the router keeps doing throughout this chapter: inferring, passage by passage, whether it faces a comparison, a process, a hierarchy, a causal claim, a chronology, a qualification-heavy argument. That structural description — currently re-derived from raw prose every time, fragilely, implicitly — has nowhere to live. There is no stable object holding “this passage compares four options across eleven dimensions with these values”; there is only the prose, read again, guessed at again.

The chapter does not solve this. It deliberately gives Chapter 5 no explicit claims/relations/evidence schema — no ledger of atomic claims, no relation edges, no provenance links. The router works from informal structure judgements, and the reader should feel the repeated cost: every enhancement decision re-infers what the previous one already discovered, and nothing accumulates. That pain is the point. It produces the next problem naturally: if many representations are possible, what stable object are they representations of? Chapter 6 earns its machinery — the semantic record — as the answer to a need this chapter makes concrete rather than as a formalism introduced on authority.

What this chapter earned

Understanding, unlike selection, can be improved by adding a representation — but only when communicative structure, affordance fit, and benefit-minus-risk all agree, with UNCHANGED as a successful output rather than an admission of defeat. Routing errors and generation errors are distinguished; structure does the routing; Mayer’s principles constrain rather than decorate; DiagramEval prices generation risk; and the repeated re-inference of structure creates the hunger Chapter 6 feeds.

Multiple faithful views require an object underneath the page that is not identical to its prose.

References

• DiagramEval: Evaluating LLM-Generated Diagrams via Graphs (EMNLP 2025, 2025.emnlp-main.640; preprint arXiv:2510.25761; code github.com/ulab-uiuc/diagram-eval). Diagrams-as-graphs; node vs path alignment; SOTA models weak on relationships; CLIPScore layout-sensitivity. Used for generation-risk pricing and the routing/generation separation.
• Mayer, R.E., CTML principles (via Ch 02 evidence base). Reused here strictly as engineering constraints (contiguity, redundancy, coherence, signalling); no new quantitative claims taken.
• Cao, Jiang & Xia (CHI 2025). Reused for the task-driven routing lineage; UNCHANGED-as-valid-output parallels Jelly’s user-correction loop at the policy level.

Proposed experiment EXP-05: the router across four passage classes

Status: PROPOSED. Corpus: passages in four pre-registered classes — (a) sufficient prose, (b) embedded-exception prose, (c) exact-value comparison, (d) topological process — plus purely discursive controls, each with human-adjudicated structure labels and comprehension/transfer instruments. Router: proposes at most two enhancements or UNCHANGED per passage, logging structure judgement → candidate → benefit/risk estimate. Baselines: plain page; human-designed enhancement; AI enhancement without UNCHANGED allowed (ablates restraint); AI router with UNCHANGED allowed. Scoring separates routing appropriateness (was the type choice defensible for the structure? judged blind to instance quality) from instance quality (was this instance faithful and legible? DiagramEval-style node/path checks where applicable). Outcomes, reported separately: comprehension, transfer to a novel case, recall, time, perceived effort, preference. Failure criteria: router never selects UNCHANGED (no restraint); router selects UNCHANGED for class (d) topology (cowardice); enhancement wins preference while losing transfer (attractiveness confound); diagrams drop embedded exceptions at above-baseline rates (distortion confirmed — kept as a substantive negative). Artifacts expected: passage pack, structure keys, router traces, per-outcome tables. What a positive result would not justify: that the router understood the passage in any deep sense — structure judgements here are informal and unaccumulated, which is exactly the limitation Chapter 6 removes.