Communication Was Never Just Words

A junior engineer is stuck on a distributed-systems bug: two services deadlock under load, but only on Tuesdays. A senior colleague tries three explanations in sequence. First, a paragraph of prose describing lock ordering. The junior nods without understanding. Second, a hand-drawn diagram on a whiteboard: two timelines, two locks, arrows showing who waits for whom. Something clicks. Third, the senior mimes it with two coffee cups, sliding them across the table — and the junior laughs, because now it is obvious. Same information, three channels, three different outcomes. The prose was not wrong. It was the wrong carrier for that idea, for that person, at that moment.

This chapter asks what different representational channels can communicate. That question sounds innocent, and it is where a large shelf of folk theory waits to mislead us. The temptation is to answer it with a universal law — pictures beat words, or every person has one true channel — and both answers are wrong in instructive ways. Getting the conditional answer right matters for everything downstream: an AI that chooses representations needs to know when a diagram helps, when it decorates, and when it actively harms. This chapter earns the conditional answer. A different question — whether a particular person has one consistently superior channel — belongs to Chapter 22, which will take it apart properly. Here we state only the boundary: this chapter is about channels and materials, not about types of people.

The oldest interface is multimodal

Before writing existed, communication already ran on parallel tracks. Spoken language arrived wrapped in prosody, pace, gesture, gaze, posture, and shared surroundings — and the wrapping was often doing half the work. A warning shout carries its urgency in pitch and speed before a single word is parsed. A story told around a fire used rhythm and repetition as error correction for memory. Ong’s account of orality, met in the previous chapter, makes this point structurally: oral cultures developed elaborate formulaic patterning precisely because the channel was evanescent and the receiver’s memory was the only storage.

Writing stripped most of that away and compensated with new machinery: word order, punctuation, headings, lists, tables, illustrations in the margins. Print added typography, layout, page numbers, indexes — a whole visual addressing system for navigating a text. None of this is decoration. A timetable, a circuit schematic, a subway map, and a knitting pattern are all cases where the spatial arrangement is the content, and rendering them as running prose would be perverse. Try describing the London Underground as a sequence of sentences and you will feel, physically, the cost of using the wrong channel.

So the book’s starting observation is deliberately unoriginal: human meaning routinely travels across several channels at once — verbal, pictorial, spatial, gestural, typographic — and the channels interact. The question is when combining them helps, when it hurts, and why. For that we need mechanisms, not slogans.

Three mechanisms worth keeping

Dual coding. Allan Paivio’s dual-coding theory (1971, 1986) proposes separate but interconnected verbal and imaginal systems: a picture of a dog tends to evoke its name, giving two traces where a word alone often gives one. The classic prediction — pictures remembered better than words — has genuine support. Shepard (1967) showed recognition accuracy of 96.7% for studied pictures against 88.4% for studied words; Standing, Conezio and Haber (1970) had participants recognise over 2,000 studied pictures at above 90% accuracy days later. This is a real phenomenon, and the book will use it. But note its exact shape: it is about memory for items under particular test conditions, not about understanding complex material, and even within memory research it has hard boundaries, examined below. Dual coding explains why a labelled diagram can stick; it does not license “visuals are better” as a general law.

Cognitive load and multimedia learning. Richard Mayer’s Cognitive Theory of Multimedia Learning (CTML), developed across decades and summarised in the Cambridge Handbook of Multimedia Learning, starts from three assumptions: separate channels for visual and verbal material (after Paivio), strictly limited capacity in each channel (after Baddeley and Sweller), and active processing — learners must select, organise, and integrate. From this follow design principles with unusually good empirical backing for education research. Mayer’s own summaries report median effect sizes around d = 0.86 for coherence (exclude extraneous material, supported in 18 of 19 tests), d = 0.70 for signalling (highlight organisation, 26 of 28 tests), d = 0.82 for spatial contiguity (place words near the part of the graphic they describe, 9 of 9), and d = 1.31 for temporal contiguity (present narration with the animation, not after it, 8 of 8). Independent meta-analyses point the same way: a signalling meta-analysis of 103 studies (N = 12,201) found mean gains of g = 0.53 for retention and g = 0.33 for transfer, with reduced cognitive load; spatial-contiguity meta-analyses find retention gains around d = 0.48 and transfer around d = 0.39. A 2025 meta-analysis of Mayer’s own corpus (92 articles, 181 studies, 591 effects) found an overall g = 0.37 — real, moderated by nearly everything, and, as Mayer himself noted in a 2024 reflection, increasingly a call for boundary conditions rather than blanket endorsement.

That last clause is the one this book keeps. CTML is not permission to add pictures. Half its principles are about removing things: coherence punishes seductive details, redundancy warns against narrating on-screen text verbatim, contiguity punishes separated layouts. The theory predicts, correctly, that a diagram helps when it offloads structure the prose makes laborious, and hurts when it duplicates, decorates, or splits attention. The mechanism is capacity management, not magic.

Signalling and contiguity as attention instruments. Within CTML, two principles deserve special status because later chapters reuse them as engineering tools. Signalling — headings, emphasis, arrows, highlighting, spoken stress — guides selection: it tells limited capacity where to spend itself. Spatial and temporal contiguity — keeping related words and pictures together in space and time — reduce the integration cost of building one mental model from two inputs. When the Semantic Browser (Chapter 10) places a claim beside its evidence, or the sidecar (Chapter 7) highlights the sentence a retrieved paper bears upon, they are applying signalling and contiguity, not inventing them. The book inherits these; it does not rediscover them.

Where pictures fail: the boundaries that make the theory honest

A mechanism without stated failure modes is a slogan. Three boundaries keep this chapter honest, and each becomes a design constraint downstream.

Boundary 1: retrieval governs the advantage. Weldon and Roediger (1987) reversed the classic picture-superiority effect by changing only the test. Free recall favours pictures; word-fragment completion favours studied words, because the test’s surface demands overlap with the study format — transfer-appropriate processing. The moral generalises: there is no context-free “picture memory advantage”. There is an encoding–retrieval match, and whoever designs the evaluation chooses the winner unless conditions are predeclared. Every experiment in this book therefore fixes its outcome measures — comprehension, recall, transfer, time — before running, and expects them to disagree.

Boundary 2: distinctiveness, not pictureness. Ensor, Surprenant and colleagues (2019) made words physically distinctive — varying font, size, colour, capitalisation — and pictures uniform (black-and-white), and the picture-superiority effect in recognition shrank to nothing; making the pictures similar objects at similar orientations reversed it. A 2024 extension carried the result into associative recognition and free recall and concluded dual coding is “no longer a viable explanation of the picture-superiority effect” specifically — while explicitly preserving dual coding’s value elsewhere. The lesson for this book: some of what looks like a deep fact about images is a fact about variability. A wall of uniform generated thumbnails may carry less mnemonic distinctiveness than one well-set paragraph. Anyone building visual previews (Chapter 4) should read that sentence twice.

Boundary 3: shared labels destroy the advantage. When studied pictures share verbal labels with test foils — many cats at study, a different cat at test — the picture advantage collapses, because the automatically generated label (“cat”) matches everywhere and only the image code discriminates (the conceptual fan effect; cf. Reder and colleagues, 2006). Translated to our problem: a diagram whose elements all look alike, or a visual language the reader has not learned, adds load without adding discrimination. Visual literacy is learned, domain by domain — circuit symbols, map projections, box plots — and a representation the receiver cannot decode is noise with nice colours.

Together these boundaries convert “pictures help” into the working hypothesis the book actually needs: a well-chosen second channel helps when it carries structure the first channel makes expensive — sequence, comparison, hierarchy, spatial relation, causal direction — to a receiver equipped to decode it, under conditions where the test rewards the structure carried. That sentence is doing a lot of work. Chapters 4 and 5 will spend it.

Preference is not performance

One more separation before we leave the general theory, because AI systems constantly conflate the two. People reliably prefer illustrated, video, or interactive versions of material — and then often learn no more, or learn less, while feeling they learned more. Fluency masquerades as understanding: the smooth animation, the confident narration, and the attractive layout raise judgements of learning without raising transfer. The book’s evaluation discipline therefore always measures preference and performance separately and reports their disagreements. A representation people love that teaches nothing is a failure with good ratings. A representation people find effortful that reliably transfers is a candidate worth refining. Chapter 5’s enhancement router will be scored exactly this way, and Chapter 22 will show why mistaking the first for the second built an industry of personalisation theatre.

Closely related is the fixed-labels trap, stated here as a boundary and demolished in Chapter 22. The best current evidence on matching instruction modality to self-reported learning styles — Clinton-Lisell and Litzinger’s 2024 meta-analysis (21 studies, 101 effect sizes, N = 1,712) — found a small overall benefit (g ≈ 0.31) that mostly evaporates under the criterion that actually matters: only about 26% of outcome measures showed the crossover interaction (each group benefiting from its own matched form) that Pashler and colleagues (2008) set as the bar, study quality was generally low, and the authors themselves concluded the benefits are too small and infrequent to warrant adoption, warning about essentialism and the exclusion of unclassifiable students. This book’s position follows the authors, not the headline number: do not sort people into visual, auditory, or kinesthetic kinds; do condition representations on task, material, goal, and context — with accessibility needs as genuine, specific exceptions, not personality types. The full argument, with the crossover logic explained, belongs to Chapter 22. Nothing in this chapter licenses person-typing, and any representation policy built from this chapter alone must offer every reader every channel.

Representation cost versus communication gain

Every added channel has a price, paid in at least four currencies. Production cost: someone or something must make the diagram, and generation is cheap while correctness review is not. Attention cost: the receiver must decode a second channel and integrate it with the first — split attention is a real tax. Distortion risk: each transformation can drop the exception, flip the relation, or smooth the uncertainty (Chapter 9 will build the instrument that prices this). Opportunity cost: space and time given to the visual are taken from prose, evidence, or nothing — and nothing is sometimes the right answer.

The book’s decision rule, first stated here and applied from Chapter 4 onward: add a representation only when its expected communication gain — for this material, this task, this receiver — exceeds its total cost including distortion risk, and keep UNCHANGED on the table as a candidate that often wins. One scoping rule governs what “gain” means in this chapter and the next: the licensed outcomes are decision quality and decision speed (Chapter 4) and task-bounded comprehension and transfer (Chapter 5) — never retention, never durable learning. Representational affordance, immediate comprehension, retention, and transfer are four different rungs; evidence on one never licenses claims on the next without its own measures. This is why the fixed page, examined next, is best understood not as a relic but as one point in a design space: author-time rendering with zero marginal decision cost and maximal predictability. Everything the book proposes must beat that baseline on a predeclared measure, not merely differ from it.

What this chapter earned

Meaning travels across channels — verbal, pictorial, spatial, typographic, gestural — and combining channels helps exactly when the second channel carries structure the first makes expensive, to a receiver who can decode it, under evaluation that rewards that structure. Dual coding, cognitive-load management, signalling, and contiguity are the mechanisms; retrieval match, distinctiveness, shared-label interference, and the preference–performance gap are the boundaries. No universal visual advantage was found, and none is claimed. No person was sorted into a type, and none will be until Chapter 22 takes up that question directly.

Once communication can take multiple forms, the fixed page becomes an engineering choice.

References

• Mayer, R.E. (2021). Research-based principles for designing multimedia instruction. Summary of CTML assumptions (dual channels; limited capacity; active processing) and principles with median effect sizes: coherence d ≈ 0.86, signalling d ≈ 0.70, spatial contiguity d ≈ 0.82, temporal contiguity d ≈ 1.31. Used here for the mechanism and its direction, not as a guarantee for any single design.
• Richter, J., Scheiter, K. & Eitel, A. (2016; meta-analysis reported via Schneider et al. 2018 and related reviews). Signalling meta-analysis: 103 studies, N = 12,201; retention g ≈ 0.53, transfer g ≈ 0.33; cognitive load reduced. Cited for the signalling effect size and its transfer attenuation.
• Clinton-Lisell, V. & Litzinger, C. (2024). Is it really a neuromyth? A meta-analysis of the learning styles matching hypothesis. Frontiers in Psychology, 15, 1428732. DOI 10.3389/fpsyg.2024.1428732. 21 studies, 101 effect sizes, N = 1,712; overall g ≈ 0.31 but crossover interaction in only ~26% of measures; low study quality; authors conclude against widespread adoption. Used for the fixed-labels boundary.
• Weldon, M.S. & Roediger, H.L. (1987). Altering retrieval demands reverses the picture superiority effect. Memory & Cognition. Free recall favours pictures; word-fragment completion favours studied words. Used for the retrieval-match boundary.
• Ensor, T.M., Surprenant, A.M. et al. (2019). Increasing word distinctiveness eliminates the picture superiority effect in recognition. Memory & Cognition. Distinctive words vs uniform pictures abolish the effect; reversal with similar-orientation pictures. Used for the distinctiveness boundary. Extended in Q. J. Exp. Psychol. (2024) to associative recognition and free recall.
• Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford University Press. Source of the dual-coding mechanism as used here.
• Pashler, H. et al. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest. Used only as the methodological standard (crossover interaction) that later work, including Clinton-Lisell & Litzinger, adopts. Full demolition deferred to Chapter 22.

Proposed experiment EXP-02: modality × task

Status: PROPOSED. Hypothesis: the best modality is conditional on material and task, not universal. Independent variable: form (text-only / image-only / text+image with signalling+contiguity / diagram-assisted). Include pre-registered cases where the visual should not help (purely discursive argument, shared-label materials). Control/baseline: text-only. Population: adults with stated domain familiarity recorded, not sorted into types. Measurements (fixed before running): comprehension, delayed recall, transfer to a novel case, time on task, and preference — reported separately. Failure criteria: one form dominates all materials on all measures (collapses conditionality); or preference and transfer disagree with no account (forces revision of the cost model). Artifacts expected: material pack, scoring rubrics, per-measure results table. What a positive result would not justify: any claim about kinds of people; person-level conditioning belongs to EXP-22/EXP-24.