# Communication Was Never Just Words
# Book: Language
# Chapter URL: https://programmer.ie/books/language/03-chapter/
# Prose fingerprint (sha256, normalized): b77e7ddc6ae2dec1a516405cf263b91d9355f06065f761c36914c36485fe74b1
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A junior engineer is stuck on a distributed-systems bug: two services deadlock during a batch process that runs on Tuesdays. A senior colleague tries three explanations in sequence. First, a paragraph describes lock ordering. The junior nods, but cannot say which service is waiting for which lock. Second, the colleague draws two timelines on a whiteboard: service A holds lock one and waits for lock two; service B holds lock two and waits for lock one. The circular wait becomes visible. Third, the colleague uses two coffee cups as the locks and acts out the sequence. Now the junior can explain why neither service can move.
It is tempting to say the diagram and demonstration succeeded where prose failed. They may have helped; they also came after two earlier explanations, and the senior had another chance to respond to confusion. This is an illustration, not an experiment isolating the effect of format. But it exposes the design question: which relationships did each representation make available, and what could the receiver reconstruct and use?
If a receiver reconstructs meaning from the evidence a representation supplies, its form matters. Prose can express lock order exactly; a timeline can make simultaneous waiting easier to inspect. The representations need not contain different facts to impose different reconstruction work. This chapter asks when a change of form, or the addition of another form, helps a particular receiver perform a particular task. It rejects two shortcuts: pictures always beat words, and each person has one true medium. A system that selects representations needs a more demanding answer: what obligation must the form preserve, what benefit should it produce, and how would we discover that it did not?
One boundary first. Channel, modality and representation are related but not interchangeable. A channel is a route through which communication reaches someone; a modality concerns how it is perceived, such as hearing or sight; a representation is the organisation of what is conveyed, such as a timeline, map or paragraph. A written paragraph and a diagram may both be visual, while speech can combine words, prosody and gesture. This chapter sometimes uses channel in the ordinary broad sense, but the engineering decision ahead is usually about representation. None of these distinctions licenses sorting readers into fixed visual, auditory or kinaesthetic types. A person’s accessibility requirements are real constraints; a supposed lifelong learning-style label is a different claim requiring different evidence.
The oldest interface is multimodal Before writing existed, communication already combined spoken words with prosody, pace, gesture, gaze, posture and shared surroundings. A warning can be made urgent by pitch as well as wording. Repeated rhythms and formulas can help a story remain recognisable when it is retold. Walter Ong’s account of oral traditions draws attention to the demands of communication without a fixed written record. But it would be a mistake to reduce oral cultures to one listener’s memory: communities also preserve knowledge in shared practices, performance, material arrangements and repeated social exchange. The defensible point is about available means of preservation and correction, not about an absence of sophistication.
A written document can travel without the speaker’s voice, gaze or immediate surroundings. It also affords other resources: revision, spatial arrangement, punctuation, headings, lists, tables, diagrams and marginal notes. Writing did not invent word order, and neither writing nor print automatically guaranteed reliable meaning; they changed what could be held still, compared and revisited. Print further widened the reach of typography, pagination, indexes and standardised layouts. A timetable, circuit schematic, subway map and knitting pattern illustrate why arrangement is sometimes part of the information itself. Running prose can describe a route through the London Underground, but a map can make alternative paths and connections available for comparison at once.
The starting observation is not that images replaced words. Communication already draws on verbal, pictorial, spatial, gestural and typographic resources, often in combination. Some describe the same fact in different ways; others contribute complementary information. That distinction matters: replacing prose with a diagram and placing a diagram beside prose are different interventions, with different attention and preservation costs. The question is what each intervention lets a receiver recover, and what it obscures.
Why form changes reconstruction Dual coding. Allan Paivio’s theory (1971, 1986) proposes interacting verbal and nonverbal representational systems. A picture may invite both a visual representation and a verbal label, but these are theoretical mechanisms, not two guaranteed traces for every image. Classic recognition experiments support an advantage for pictures under certain conditions: Shepard (1967) compared recognition for words, sentences and pictures; Standing, Conezio and Haber (1970) studied recognition after exposure to 2,500 visual stimuli. Those results concern recognition memory for studied items, not a general advantage in explanation, causal reasoning or transfer. They make dual coding a plausible starting hypothesis for a labelled diagram, not evidence that a diagram will teach a given concept better. Later findings on retrieval demands and distinctiveness narrow the claim further.
Cognitive load and multimedia learning. Richard Mayer’s cognitive theory of multimedia learning starts with partly distinct processing channels, limited processing capacity and active learning: learners select, organise and integrate material. Visual and auditory presentation are not identical to pictorial and verbal content; on-screen words and a diagram both make demands on visual attention. The theory motivates design principles such as coherence (exclude extraneous material), signalling (make organisation visible), spatial contiguity (place related words and pictures together) and temporal contiguity (coordinate narration with animation). Their effects are not uniform. Richter, Scheiter and Eitel’s 2016 meta-analysis of text–picture signalling found benefits that depended in part on learners’ prior knowledge. Cromley and Chen’s 2025 meta-analysis of Mayer’s research programme, covering 181 studies and 591 effects, reported a pooled Hedges’ g of 0.37, with substantial moderation by principle, medium, age, domain and outcome. It also found stronger effects for some active-learning interventions than for presentation changes alone. The useful conclusion is conditional: choose a form for the processing problem it addresses, not because a pooled effect makes multimedia a universal upgrade.
That last point is worth keeping. Multimedia principles are not instructions to add more material. Coherence can favour removing seductive details; redundancy warns that simultaneous on-screen text and identical narration may compete rather than reinforce; contiguity concerns the work of connecting related elements. These are testable design hypotheses, not automatic verdicts. A diagram may externalise a causal relationship, while an unnecessary animation may force a reader to divide attention or wait for information a static figure shows immediately. The form has to be judged against the task and the receiver’s observable performance.
Signalling and contiguity as attention instruments. Two principles become practical controls in the system built later. Signalling can direct attention with headings, arrows, highlighting or spoken stress. Contiguity can reduce the work of connecting a claim with the relevant part of a figure or source. A Semantic Browser might display a summary assertion beside the exact passage from which it was derived, highlighting the passage on request. That arrangement can make checking easier. It does not itself establish that the assertion is true, that the passage supports it, or that the reader checked it. Placement is an interface property; source support and successful reconstruction require separate tests.
Conceptual comparison. Multimodality is not a quantity of channels; it is a question about which evidence the task needs. Open full-size diagram. Where pictures fail A mechanism without stated failure modes is a slogan. Three boundaries keep this one honest, and each becomes a design constraint later.
Retrieval demands change the advantage. Weldon and Roediger (1987) studied pictures and words under different tests. Pictures did better in free recall, while studied words produced more priming on word-fragment completion; when the test used degraded-picture identification, the balance changed again. This is not a claim that every difference in memory comes from test format alone. It is evidence for a narrower principle: performance depends partly on the match between operations during study and those required at test. For our purposes, a representation experiment must name its outcome before judging a winner. Immediate recall, causal explanation, delayed retention, task completion and transfer can legitimately disagree.
Distinctiveness can matter more than pictureness. Ensor, Surprenant and Neath (2019) varied word font, size, colour and capitalisation across experiments; in one experiment they also reduced pictorial distinctiveness by using black-and-white images with similar orientations. Under their recognition-test conditions, making words more distinctive could eliminate the familiar picture advantage. This challenges an explanation that credits the advantage to image format alone. It does not establish that unusual typography generally improves learning, or that uniformly styled diagrams generally fail. Distinctiveness is one candidate mechanism to isolate, not a design command to make everything look different.
Labels can conceal the distinction the task requires. Consider a test where a reader must distinguish two similar components that share the label gateway, or tell whether service A waits for service B rather than the reverse. A diagram whose boxes all look alike may make the category clear while leaving the decisive relation hard to detect. An unfamiliar legend can have the same effect. This is a design risk, not a separately established picture-superiority result from the studies cited here. Its consequence for the Semantic Browser is concrete: the correctness of entity identity, direction, order, uncertainty and exceptions must be checked against the source; neither a familiar icon nor a tidy layout supplies that evidence.
Together these boundaries turn “pictures help” into the working hypothesis the book needs:
An alternative or additional representation helps when it preserves the required claims and relationships, makes a task-relevant distinction less costly to recover, and can be understood by this receiver under these conditions.
That is a proposed decision rule, not a result of an experiment in this book. It contains at least three questions that must be answered separately: was the transformation faithful to its source, could the receiver use it, and did the new form improve on a simpler alternative? A format can pass any one while failing the others. The next chapters will need all three answers before promoting a generated representation.
Preference is not performance One more separation, because generated interfaces make it easy to blur. People can prefer a polished illustration, animation or interactive view while performing no better on the intended task; the reverse can also occur. Ease of reading, confidence in understanding, immediate recall and successful transfer are different outcomes. The chapter does not establish how often preference and performance diverge, and a rating alone cannot settle it. Evaluate preference and task performance separately, including time and errors, and preserve disagreement rather than averaging the measures into one impression. A view readers love but cannot use needs revision; a demanding view that improves transfer may be worth retaining, but may still need work on accessibility and effort.
Closely related is the fixed-labels trap. Clinton-Lisell and Litzinger’s 2024 meta-analysis of learning-style matching examined 21 eligible studies and 101 effect sizes. It reported a small pooled matched-instruction advantage (g = 0.31), but only 26% of learning-outcome measures showed the crossover pattern needed to support the stronger claim that different self-identified groups learn best from their own matched form. The studies were generally of low quality, and the authors judged the benefits too inconsistent to justify widespread matching. That is not evidence that the choice of representation never matters; it rejects promotion of fixed style labels into an instruction policy. Condition on task, material, goals and observed needs, not supposed person types. Accessibility requirements should be met as constraints, not treated as optional gains to trade away against cost. Where feasible, let readers choose a form, request an alternative and override an inferred preference; a single successful interaction should not become a permanent personal classification.
An experiment the Semantic Browser can run The opening deadlock example cannot identify a causal format effect because the explanations were sequential. Here is a proposed test that could. Freeze one short source describing the two-lock deadlock, including which service holds and waits for each lock, why the Tuesday batch exposes the problem, and the correction being considered. Declare those facts and relations as preservation obligations before anyone generates a view. They are not optional decoration.
Prepare three source-linked conditions: the original prose alone; a diagram with the same facts and explicit lock-direction arrows; and prose plus diagram together. A decorative diagram with no additional useful structure could serve as a negative-control condition, provided it is also checked for faithfulness. Have an independent reviewer check every generated view for omitted obligations, reversed arrows and unsupported additions before readers see it. A faithful diagram is an eligible candidate, not yet a demonstrated improvement.
Give readers equivalent, counterbalanced tasks so that exposure order and repeated explanation do not decide the result. Predetermine a primary outcome such as correctly diagnosing which service must release or reorder a lock, then record secondary outcomes: error type, response time, confidence, preference and later transfer to a different two-resource deadlock. Record prior domain knowledge and accessibility requirements without assigning learning-style types. Compare each arm against the unchanged source; report uncertainty, and allow a result of no gain or harm. This protocol has not been run here. It specifies what evidence the claim “the diagram helped” would require, and it separates source fidelity from receiver performance.
What a second channel costs Every additional representation has a price, paid in several currencies. Production and checking: a diagram must be generated or drawn, then checked for reversed arrows, omitted exceptions and invented causal links; cheap generation does not make faithful generation free. Attention and time: a receiver may need to decode a legend, switch between prose and diagram, or wait through an animation. Operational cost: dynamic rendering consumes computation and can add latency. Opportunity cost: screen space and reading time spent on one form are unavailable for another, and sometimes the unchanged source is enough. Some obligations are not ordinary costs. If a transformation drops a required safety condition or makes the material inaccessible, its apparent speed or popularity cannot compensate; it fails an eligibility test before competing on usefulness.
The decision rule for the rest of the book therefore has two stages. First, refuse representations that violate the declared preservation obligations or the receiver’s access requirements. Then, among admissible choices, compare task benefit with generation, checking, attention and operating cost, keeping the unchanged source as a baseline. This is a design rule to evaluate, not a proven universal utility function. Gain has the meaning established in Chapter 2: the underlying information becomes easier to recover, compare or act on. A form that makes a relation visible, one that improves an immediate answer, one remembered a day later, and one transferable to a new case have earned four different claims. Neither a similarity score nor a preference rating can collapse them into one. If the evidence cannot distinguish the alternatives, the system may retain the existing representation or ask what the receiver needs.
This puts the fixed page in its true light. It is not a relic. An author has already paid the cost of choosing, arranging and reviewing it; the reader need not pay for a new rendering decision on every encounter. A fixed page is not costless or infallible, but it is a meaningful baseline with a stable artifact to inspect. A dynamic representation should earn its place against that baseline for the declared task, with the same underlying source, rather than by looking newer or more elaborate.
Documented trade-offs, stated as a balance rather than a verdict. The chapter does not license matching readers to representational types. Open full-size diagram. Where this leaves us Communication draws on verbal, pictorial, spatial, typographic and gestural resources, and each may make different evidence available for reconstruction. Dual coding and multimedia-learning research provide hypotheses about why some combinations help; retrieval demands, distinctiveness and task design show why the result cannot be universal. The receiver’s preference is not their performance, and representation choice is not a diagnosis of the receiver’s personality. For a source-linked system the engineering test is now clearer: preserve what matters, compare what the receiver can do, measure the added cost, and leave the unchanged source available. The proposed deadlock comparison would test that claim; this chapter does not report a completed experiment.
So far, the examples have been forms chosen in advance by an author, teacher or colleague. The next step is to let the representation be chosen or revised at use time, for a stated task, without pretending that a new rendering is a new source of truth. That makes provenance, preservation, receiver choice and the option to keep the original page part of the interface. Later chapters will ask when the system is allowed to use personal context to make that choice. First we have to examine what happens when the supposedly fixed page becomes dynamic.
References Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart and Winston. Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford: Oxford University Press. Shepard, R.N. (1967). Recognition memory for words, sentences, and pictures. Journal of Verbal Learning and Verbal Behavior, 6(1), 156–163. Standing, L., Conezio, J. & Haber, R.N. (1970). Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Science, 19(2), 73–74. Mayer, R.E. (2021). Multimedia Learning (3rd ed.). Cambridge: Cambridge University Press. Cited for the theoretical assumptions and design principles. Richter, J., Scheiter, K. & Eitel, A. (2016). Signaling text–picture relations in multimedia learning: A comprehensive meta-analysis. Educational Research Review, 17, 19–36. Cromley, J.G. & Chen, R. (2025). A meta-analysis of Richard Mayer’s multimedia learning research: Searching for boundary conditions of design principles across multiple media types. Educational Research Review, 49, 100730. DOI 10.1016/j.edurev.2025.100730. Weldon, M.S. & Roediger, H.L. (1987). Altering retrieval demands reverses the picture superiority effect. Memory & Cognition, 15(4), 269–280. Ensor, T.M., Surprenant, A.M. & Neath, I. (2019). Increasing word distinctiveness eliminates the picture superiority effect in recognition: Evidence for the physical-distinctiveness account. Memory & Cognition, 47, 182–193. Clinton-Lisell, V. & Litzinger, C. (2024). Is it really a neuromyth? A meta-analysis of the learning styles matching hypothesis. Frontiers in Psychology, 15, 1428732. DOI 10.3389/fpsyg.2024.1428732. Pashler, H., McDaniel, M., Rohrer, D. & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119. Ong, W.J. (1982). Orality and Literacy: The Technologizing of the Word. London: Methuen.