Communication Was Never Just Words

A junior engineer is stuck on a distributed-systems bug: two services deadlock under load, but only on Tuesdays. A senior colleague tries three explanations in sequence. First, a paragraph of prose describing lock ordering. The junior nods without understanding. Second, a hand-drawn diagram on a whiteboard: two timelines, two locks, arrows showing who waits for whom. Something clicks. Third, the senior mimes it with two coffee cups, sliding them across the table, and the junior laughs, because now it is obvious. Same information, three channels, three different outcomes. The prose was not wrong. It gave this person, at this moment, the wrong evidence to build from.

If a receiver reconstructs meaning from whatever a representation puts in front of them, then the form of the representation matters, because form decides what evidence the receiver has. This chapter asks what different channels can give a receiver. The question sounds innocent, and a large shelf of folk theory waits behind it. The temptation is to answer with a universal law (pictures beat words, or every person has one true channel), and both answers are wrong in instructive ways. The right answer is conditional, and getting it right matters for everything that follows. A system that chooses representations has to know when a diagram helps, when it only decorates, and when it actively harms.

One boundary first. This chapter is about channels and materials: what a form makes possible for a given task and receiver. It is not about kinds of people. Whether a particular person has one consistently best channel is a separate question, and a later chapter takes it apart. Here the claim is only that nothing in what follows licenses sorting readers into visual, auditory or kinaesthetic types.

The oldest interface is multimodal

Before writing existed, communication already ran on parallel tracks. Spoken language arrived wrapped in prosody, pace, gesture, gaze, posture and shared surroundings, and the wrapping often did half the work. A warning shout carries its urgency in pitch and speed before a single word is parsed. A story told around a fire used rhythm and repetition as error correction for memory. Walter Ong’s account of oral cultures makes the structural point: the formulaic patterning of oral tradition developed because the channel was fleeting and the listener’s memory was the only storage.

Writing stripped most of that away and compensated with new machinery: word order, punctuation, headings, lists, tables, illustrations in the margins. Print added typography, layout, page numbers and indexes, a whole visual addressing system for finding your way through a text. None of this is decoration. A timetable, a circuit schematic, a subway map and a knitting pattern are all cases where the spatial arrangement is the content, and rendering them as running prose would be perverse. Try describing the London Underground as a sequence of sentences and you will feel, physically, the cost of the wrong channel.

So the starting observation is deliberately unoriginal. Human meaning routinely travels across several channels at once (verbal, pictorial, spatial, gestural, typographic), and the channels interact. The questions are when combining them helps, when it hurts, and why. For that we need mechanisms, not slogans.

Three mechanisms worth keeping

Dual coding. Allan Paivio’s theory (1971, 1986) proposes separate but connected verbal and imaginal systems: a picture of a dog tends to evoke its name, leaving two traces where a word alone often leaves one. The classic prediction, that pictures are remembered better than words, has real support. Shepard (1967) found recognition for studied pictures higher than for studied words, and Standing, Conezio and Haber (1970) found remarkably high recognition after a single viewing of a very large set of pictures, about 2,500 according to the title of their report. But look at its exact shape. It is a finding about memory for items under particular test conditions, not about understanding complex material, and even within memory research it has hard boundaries, taken up below. Dual coding explains why a labelled diagram can stick. It does not license “visuals are better” as a general law.

Cognitive load and multimedia learning. Richard Mayer’s cognitive theory of multimedia learning starts from three assumptions: separate channels for visual and verbal material (after Paivio), strictly limited capacity in each (after Baddeley and Sweller), and active processing, in which learners select, organise and integrate. From these follow design principles with unusually good empirical backing for education research. Mayer’s own summaries report moderate to large median effects for principles such as coherence (leave out extraneous material), signalling (highlight the organisation), spatial contiguity (put words next to the part of the graphic they describe) and temporal contiguity (present narration with the animation, not after it), though the figures differ between compilations. Independent evidence points the same way with more modest numbers. A meta-analysis of signalling in text and picture material (Richter, Scheiter and Eitel, 2016) found a small to medium benefit, strongest for learners with little prior knowledge. And a 2025 meta-analysis of Mayer’s own research programme (Cromley and Chen) found an overall effect of about 0.37, real but moderated by nearly everything, and framed its task as searching for boundary conditions of the design principles.

That last point is the one worth keeping. The theory is not permission to add pictures. Half its principles are about removing things: coherence punishes seductive details, redundancy warns against narrating on-screen text word for word, contiguity punishes separated layouts. It predicts that a diagram helps when it offloads structure the prose makes laborious, and hurts when it duplicates, decorates or splits attention. The mechanism is management of limited capacity, not magic.

Signalling and contiguity as attention instruments. Two of those principles deserve special status, because the rest of the book uses them as engineering tools. Signalling (headings, emphasis, arrows, highlighting, spoken stress) tells limited capacity where to spend itself. Contiguity (keeping related words and pictures together in space and time) lowers the cost of building one model from two inputs. When a system later places a claim beside its evidence, or highlights the sentence a retrieved paper bears on, it is applying signalling and contiguity, not inventing them.

Where pictures fail

A mechanism without stated failure modes is a slogan. Three boundaries keep this one honest, and each becomes a design constraint later.

Retrieval governs the advantage. Weldon and Roediger (1987) reversed the classic picture-superiority effect by changing only the test. Free recall favours pictures, but word-fragment completion favours studied words, because the demands of the test overlap with the format of study. The moral generalises. There is no context-free “picture memory advantage”, only a match between how something was encoded and how it is retrieved, and whoever designs the evaluation chooses the winner unless the outcomes are fixed in advance. Any fair study of representations must name its outcome measures (comprehension, recall, transfer, time) before running, and expect them to disagree.

Distinctiveness, not pictureness. Ensor, Surprenant and colleagues (2019) made words physically distinctive, varying font, size, colour and capitalisation, and made pictures uniform and black and white. The picture-superiority effect in recognition was eliminated. Some of what looks like a deep fact about images is a fact about variability. A wall of uniform generated thumbnails may carry less memorable distinctiveness than one well-set paragraph.

Shared labels destroy the advantage. When studied pictures share a verbal label with the test foils (many cats at study, a different cat at test), the picture advantage collapses, because the automatically generated label matches everywhere and only the image itself discriminates. Translated to communication: a diagram whose elements all look alike, or a visual language the reader has not learned, adds load without adding discrimination. Visual literacy is learned domain by domain (circuit symbols, map projections, box plots), and a representation the receiver cannot decode gives them noise with nice colours.

Together these boundaries turn “pictures help” into the working hypothesis the book needs:

A well-chosen second channel helps when it carries structure the first channel makes expensive (sequence, comparison, hierarchy, spatial relation, causal direction), to a receiver equipped to decode it, under conditions where the task rewards that structure.

That sentence does a great deal of work, and the next several chapters spend it.

Preference is not performance

One more separation, because AI systems constantly blur it. People reliably prefer illustrated, video and interactive material, and then often learn no more, or less, while feeling they learned more. Fluency masquerades as understanding: the smooth animation, the confident narration and the attractive layout raise judgements of learning without raising transfer. This is the receiver’s overconfidence from the previous chapter, now produced by the representation itself. So any honest evaluation of a representation measures preference and performance separately and reports where they disagree. A representation people love that teaches nothing is a failure with good ratings. A representation people find effortful that reliably transfers is a candidate worth refining.

Closely related is the fixed-labels trap. The best current evidence on matching instruction to self-reported learning styles is a 2024 meta-analysis, which found a small aggregate benefit that mostly disappears under the criterion that matters: that each group do better with its own matched form. Study quality was generally low, and the authors themselves concluded the benefits were too small and too infrequent to justify adoption, warning against essentialism and against excluding students who do not fit a category. The position this book takes follows the authors, not the headline number. Do not sort people into visual, auditory or kinaesthetic kinds. Do condition representations on task, material, goal and context, treating accessibility needs as specific exceptions, not personality types. Nothing in this chapter licenses person-typing, and any policy built from it alone should offer every reader every channel.

What a second channel costs

Every added channel has a price, paid in at least four currencies. Production cost: someone or something has to make the diagram, and generation is cheap while checking it is not. Attention cost: the receiver must decode a second channel and integrate it with the first, and split attention is a real tax. Distortion risk: each transformation can drop the exception, flip the relation or smooth away the uncertainty. Opportunity cost: space and time given to the visual are taken from prose, from evidence, or from nothing, and nothing is sometimes the right answer.

The decision rule for the rest of the book follows. Add a representation only when its expected communicative gain, for this material, task and receiver, exceeds its total cost including distortion risk, and keep leaving the source unchanged on the table as a candidate that often wins. Gain here has the meaning set out in the last chapter: the same underlying information made easier to recover, compare or act on. Different kinds of gain belong to different rungs of evidence. That a form makes a relationship easy to see, that it lets a reader answer a question right now, that they remember it later, and that they can apply it to a new case are four different claims, and evidence for one never licenses the next without its own measure.

This also puts the fixed page in its true light. It is not a relic. It is one point in a design space: author-time rendering with zero marginal decision cost and maximal predictability. Anything proposed in place of it has to beat it on a measure fixed in advance, not merely differ from it.

Where this leaves us

Meaning travels across verbal, pictorial, spatial, typographic and gestural channels, and each gives the receiver different evidence to build from. A second channel helps when it carries structure the first makes expensive, to a receiver who can decode it, under evaluation that rewards that structure. Dual coding, cognitive-load management, signalling and contiguity explain why. Retrieval match, distinctiveness, shared labels and the gap between preference and performance say where the explanation stops. No universal visual advantage turned up, and none is claimed, and no person has been sorted into a type.

So far, every choice of form has been made by an author, ahead of time, for a reader who is not there. Once the form can be changed after the author has finished, by something that can also see the reader, the choice of representation becomes part of communication itself. That begins with the page everyone assumed was fixed.

References

• Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford: Oxford University Press.
• Shepard, R.N. (1967). Recognition memory for words, sentences, and pictures. Journal of Verbal Learning and Verbal Behavior, 6(1), 156–163.
• Standing, L., Conezio, J. & Haber, R.N. (1970). Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Science, 19(2), 73–74.
• Mayer, R.E. (2021). Multimedia Learning (3rd ed.). Cambridge: Cambridge University Press (edition year to be confirmed); and the Cambridge Handbook of Multimedia Learning. Cited for the principles, not for any figure.
• Richter, J., Scheiter, K. & Eitel, A. (2016). Signaling text–picture relations in multimedia learning: A comprehensive meta-analysis. Educational Research Review, 17, 19–36.
• Cromley, J.G. & Chen, R. (2025). A meta-analysis of Richard Mayer’s multimedia learning research: Searching for boundary conditions of design principles across multiple media types. Educational Research Review, 49, 100730.
• Weldon, M.S. & Roediger, H.L. (1987). Altering retrieval demands reverses the picture superiority effect. Memory & Cognition, 15(4), 269–280.
• Ensor, T.M., Surprenant, A.M. & Neath, I. (2019). Increasing word distinctiveness eliminates the picture superiority effect in recognition: Evidence for the physical-distinctiveness account. Memory & Cognition, 47, 182–193.
• Clinton-Lisell, V. & Litzinger, C. (2024). Is it really a neuromyth? A meta-analysis of the learning styles matching hypothesis. Frontiers in Psychology, 15, 1428732. DOI 10.3389/fpsyg.2024.1428732.
• Pashler, H., McDaniel, M., Rohrer, D. & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119.
• Ong, W.J. (1982). Orality and Literacy: The Technologizing of the Word. London: Methuen.