# What the System Believes About You Must Be Visible
# Book: Language
# Chapter URL: https://programmer.ie/books/language/28-chapter/
# Prose fingerprint (sha256, normalized): 98e5aa501338e26ed8910be901783b6e19cc6ff3c717b71395f4bf5b0eca87d6
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Iris opens a talk about a consensus algorithm she has not used. The channel shows its steps and a diagram, but folds the section that introduces the algorithm’s failure assumptions. She asks why. The answer is: Consensus foundations treated as already known, inferred from earlier architecture reviews. The explanation is plausible. That is precisely the danger: plausibility is not a record of which belief actually caused the omission.
Iris knows the field. But knowing the field does not establish that she knows this algorithm’s assumptions, and knowing them would not make them optional when they qualify a result she is being shown. The first defect is the unsupported extension from one protocol to another. The second is a preservation failure: even a genuinely expert reader needs the scope of the claim. A better explanation alone would not repair either defect; the view has to be checked before the assumption can be hidden.
Chapter 27 exposed the same structural problem in a different place. An adapter could believe that a service’s broad filter implements Iris’s narrow portable rule, or could mistake a setting readback for verified enforcement. Those are beliefs about a policy or service rather than about Iris herself, but they too can change what she sees. The common question is which consequential claims the system relied on, and whether the evidence and authority for each use were sufficient.
These are constructed cases. The question they expose is:
If the system’s beliefs about me or my policies can materially change communication, what must I be able to see and correct?
A belief becomes consequential through its use A belief that Iris knows consensus is not the same object as a decision to fold this explanation. The first is a claim about her; the second is an application of that claim to a source and task. A claim may be well supported within its own scope and still be unusable here. Conversely, an explanation may be missing for reasons unrelated to her expertise, such as a rendering failure or a source obligation the system never checked. Inspectability has to distinguish the stored belief, the decision that used it and the actual effect.
The distinctions from earlier chapters therefore remain active:
DECLARED what Iris supplied or accepted OBSERVED what happened, under recorded conditions INFERRED what the system concluded from that evidence Iris’s instruction to preserve implementation detail is declared policy. Her expansion of a derivation is an observation, provided the event was actually recorded rather than reconstructed from a model’s guess. The conclusion that she prefers expanded derivations is an inference. These do not become interchangeable because all three happen to favour a detailed view. A decision record must retain the origins and the distinct permissions attached to them.
An observation also needs its conditions. If Iris expanded a derivation during an explicit deep-study session, that mode is part of the event’s provenance; using the event later as evidence of a general preference would be another inference. The source event’s time, the period a conclusion is claimed to cover and the time it is used are not the same. Chapters 23 and 24’s temporal and session boundaries apply here as much as they did to memory and overlays.
Nor can an inferred claim appear as you told us. A declaration, an observation and an interpretation need distinct origins even when the interface uses the same short sentence to show them.
Provenance and consequence A visible belief must expose both its provenance and its consequence.
We think you know consensus is insufficient. Iris needs to see what the claim rests on, what scope the evidence supports, what scope the renderer applied, and which part of the view changed. Crucially, an account of this use must come from the decision trail, not merely from retrieving any relevant memory after she asks why. A compact version of the opening failure might read:
Constructed failure, not an implemented decision trace. A real explanation must cite the inputs and gates actually recorded. Open full-size diagram. This is a constructed account of a defective decision, not an implemented record schema. In particular, the line describing a decision trace is illustrative; a real system should not print it unless it has such a trace. Evidence links should go only to records Iris is permitted to inspect. If a source includes another person’s protected information, the system may need to show its type, date and authorised basis without exposing the underlying material. Provenance cannot require unlimited disclosure, and a confidence label cannot substitute for it.
The difference between supported and applied scope must remain visible. Earlier reviews can support familiarity with a particular protocol; they do not establish mastery of a different algorithm’s failure model. In this case an independent source-preservation obligation should have blocked the fold even if the expertise inference had been valid. The account therefore needs to name both defects: an unearned scope expansion and a rendering decision that violated the claim’s own qualifications.
Using a belief outside the scope supported by its evidence is itself a new inference.
scope expansion = new inference Moving from understands consensus basics in architecture discussions to does not need introductory explanations for a different algorithm expands the belief and makes a new judgement about its use. That move needs a separate evidential basis and a separate authority check. It is not a harmless rewrite of the scope field. Even after both checks, the source-specific assumptions remain compulsory if a retained claim depends on them; expertise never grants an exemption from fidelity.
Time adds another boundary. The system should distinguish when the evidence was recorded, when the claim was considered applicable, and when the renderer used it. An inference can be a reasonable description of last month’s work and a poor basis for today’s view. A newer observation does not automatically overwrite an earlier declaration, and an old belief need not be erased to lose authority. What matters is whether this use was current, within scope and permitted.
A truthful why answer must identify the inputs and gates that actually determined the view. A fluent explanation generated after the event may be a hypothesis about the system rather than a faithful account of its decision. The host therefore needs a bounded decision receipt: the view and source versions, the applicable claims and grants, the checks performed, and the disposition of the fold. If no such receipt exists, the answer is I cannot establish why this was folded, followed by a safe reconstruction—not an invented causal story. A receipt itself proves only what the instrumented path recorded; it does not prove every internal model influence has been captured.
Beliefs about a policy also need inspection Iris can inspect the text of her portable boundary and still miss how a destination interpreted it. Keep recommendations out of the feed may become hide all research content; a service acknowledgement may become a claim of verified protection. These failures are not corrected by displaying a preference profile. They require an account of the accepted rule, the adapter’s interpretation, the control requested, and the evidence for its present state, each with its own version.
The operative claims concern rule meaning, destinations, available controls and enforcement state. Their consequences are concrete: which surface is filtered, whether search remains available, and why a receipt claims success. The interpretation should point to the accepted rule and the exact control selected. A user can then challenge a translation without being asked to rewrite the rule that was correctly declared.
policy ≠ system's interpretation of policy That gives Iris two different corrections. This rule concerns recommendations, not search results rejects the adapter’s interpretation but leaves her original boundary intact. That service has not confirmed removal disputes an enforcement claim but does not reverse her revocation. The first requires a corrected mapping; the second requires a fresh check and an honest unresolved status until that check succeeds. Neither should be reported as completed protection merely because the central record changed.
Chapter 27’s distinctions remain intact. A declared rule is not its propagated copy; a confirmed setting is not verified protection. Showing the system’s belief about either must not erase the uncertainty that belongs to the underlying event.
Visible, valid and authorised Visible means consequence-linked inspectability:
visible / inspectable ≠ continuously displayed Not every belief has to be on screen. Iris does need a route from a consequential change to the claim that produced it. Beside a folded section, Why is this folded? can expose the relevant account and restore the material. For an omitted item there may be no visible object to click, so the view also needs a discoverable what was left out or what affected this view entry, limited to information Iris is allowed to know exists. Hiding a protected item cannot be undone by exposing its title in an audit list. Inspectability and access control must be designed together.
Three questions must be answered separately:
VISIBLE Can Iris inspect the claim and its use? VALID Does the evidence support it in this scope now? AUTHORISED Is this formation, retention and use permitted? visible ≠ valid ≠ authorised A claim can be visible and false, true but out of date, or applicable but unauthorised for this use. Nor is validity binary in every case: a record can contain conflicting evidence, an unresolved mapping between concepts, or a missing source version. These conditions should remain visible as uncertainty, not collapse into a number that conceals why a gate did or did not pass.
This separation lets Iris say: Yes, I understand consensus basics. No, you may not use that belief to hide this algorithm’s assumptions. She should not have to deny her expertise to retrieve information the channel removed. More importantly, the system should have preserved those assumptions before asking her: user correction is a repair path, not the first line of source fidelity.
Visibility also cannot legitimise a prohibited inference. The exclusions on inferred mood, stress, fatigue, motivation and personality state remain exclusions. Displaying we think you are stressed with an edit button does not supply permission to form that belief or simplify communication because of it.
The same holds for retention and disclosure. Permission to derive a claim, to store it, to inspect its evidence and to use it for this decision are distinct. A person may be entitled to see that a policy used an external report without being entitled to every private field in that report. A system should keep the minimum inspectable account that supports correction while enforcing the original information boundary. Strong evidence settles none of these permissions by itself.
Visibility is not authority, and editability is not permission.
An editable belief is a remedy for an error. It cannot retroactively authorise the error, and leaving a displayed belief untouched is not acceptance. The system must not make Iris’s failure to inspect a record into consent for its continued use.
Consequence should set the bar The proposed design principle is:
The greater the consequence of an inferred belief, the more stringent its validity, permitted-use and repair requirements should be. Evidence and confirmation cannot substitute for authority.
Consider three uses. A belief that Iris finds tables useful might offer an additional table while leaving the prose available. A belief that she knows a protocol might fold an explanation and conceal an assumption she needs. A belief that she wants a message sent might lead to an external act in her name. These differ not simply by confidence but by lost access, reversibility and authority. The first may be reversible presentation; the second may violate preservation; the third requires an explicit grant dealt with later in the book.
These are consequence classes, not earned thresholds. Even an additional table can impose clutter or an accessibility cost. The relevant question is what this particular use changes, what a mistake would cost, and how readily the effect can be reversed.
Stronger evidence does not supply missing authority. Confirmation of familiarity does not waive preservation requirements; no confidence score licenses suppressing a decisive assumption. Accessibility requirements remain constraints on the view. Where applicability is uncertain, preserving access and exposing the uncertainty is a proposed conservative response.
Acting or speaking for Iris requires explicit authority. An inference may support a proposal for her to consider, but even a highly confident prediction cannot become that grant. The later part of the book takes up this stronger boundary.
The design must also work when Iris does not inspect a belief. Important information cannot depend on her finding a hidden record and correcting it first. A consequential assumption should be discoverable where its effect occurs, with access to a fuller account. Making a record available somewhere is not the same as making the effect recognisable in the current interaction.
Correction must name what changes Iris says: This algorithm is new to me. Show its assumptions in this view. That is a current instruction and a correction to the scope used here. It neither asserts ignorance of all consensus theory nor creates a preference for lengthy explanations in future sessions. The system should restore the section, mark the defective use invalid, and record only the correction and retention that are authorised.
She might instead say: Stop using inferred expertise to fold assumptions in architecture reviews. That changes the permitted use within a named scope. Or she might reject a claim outright, narrow the situations in which it applies, or ask for the retained item to be forgotten. Those are different operations:
correct the claim change its scope stop a use revise a standing rule forget the retained item The response should acknowledge the chosen operation and its extent. Restoring one explanation must not create a permanent preference. Narrowing a belief’s scope must not rewrite the original observation. Revising a standing rule must not be reported as only a session change. And forget must not promise physical erasure if the implementation merely retracts the record from future use. Different operations require different completion evidence.
A forget request also raises the risk of re-derivation. If legitimately retained observations could produce the same inference again, stopping the record’s current use is not sufficient. The system should distinguish withdrawing the derived belief, blocking its re-derivation, removing retained evidence where permitted, and confirmed deletion of particular copies. It may need a minimal, purpose-limited suppression marker to honour a no-re-derivation choice, but should not retain the discarded inference wholesale under the guise of remembering not to use it. In the runnable example, /personal block is a separate action; forget alone does not establish that block.
A correction also has dependencies. If the rejected inference contributed to two folded explanations and a cached digest, each known use has to be invalidated before that result can safely be presented again. The repair should follow direct and transitive edges in the recorded dependency graph and mark unknown edges as an uncertainty, not as proof there are none. Locality means changing the affected belief and uses without disturbing unrelated, authorised policy; it does not mean touching only one database row.
Those repairs have separate completion states:
record corrected ≠ future use stopped ≠ downstream state repaired ≠ past effect reversed A constructed account of a repair might say:
Belief: Corrected Future use: Stopped locally in the named scope Affected current view: Recomputed Cached representation: Replaced Derived belief: Invalidated for current use Previously delivered output: Past effect not reversible A past summary can be corrected or replaced, but Iris’s earlier reading or action cannot be undone. The repair report should separate known invalidations, successfully recomputed views, distributed destinations awaiting confirmation, and past effects that are irreversible. A system that discovers one known downstream use but cannot establish the completeness of its dependency graph must say so. All known dependants invalidated is narrower than all consequences repaired.
For portable policy the dependency graph extends into adapters, cached service settings and receipts. A corrected interpretation may require removal of an overbroad filter that is still operating. Until the destination confirms the relevant change, the central system can report interpretation corrected but not service now honours your instruction. If the destination is offline, the live effect remains unknown, and the user should not be given a clean bill of protection.
An instruction to stop a use must stop it. Merely lowering confidence while the view continues to be shaped by the same claim is not compliance. Recreating a synonymous inference to restore the rejected effect is not a new basis for permission. If Iris declines a proposed standing rule, that refusal retains the meaning established in Chapters 24 and 25: no promotion or continued learning towards that rejected update.
The inference’s past evidence need not be falsified to make these corrections. Iris really may have expanded earlier derivations. That observation can remain historically true if legitimately retained, while the standing preference inferred from it is rejected. Truth about the earlier interaction, current applicability and authority remain separate.
Why this view, and what would change without the belief? Inspection should explain missing things as well as visible ones. Iris can reasonably ask Why is this folded?, What did this boundary withhold?, Why was no alert sent? or Why does this receipt claim enforcement? These are different queries over the decision trail. When an item was never discovered, the honest answer is not a selection reason but a coverage limitation. The viewer must not conflate not retrieved, withheld under a rule, not presented and not noticed.
A useful response identifies the inputs actually consulted for the decision: source version, declared task, applicable standing policy, session overlay, preservation checks and particular belief uses. It distinguishes an input that determined the outcome from a belief merely present in storage. A list of everything known about Iris would be both distracting and potentially intrusive. The goal is a minimal sufficient decision account, not a complete profile.
One proposed aid is a what would have changed without this belief? comparison. For the talk, suspend only the expertise inference and rebuild from the same source version, task, applicable rules and rendering policy. Treat expertise as unknown rather than treating its withdrawal as evidence of ignorance. Such a comparison could reveal whether folding the section depended on that inference. It must not bypass preservation or accessibility requirements, because they apply in both versions.
This comparison needs an explicit limit. A freshly generated alternative might differ because of model randomness, retrieval order, prompt changes or a newer source. If those differ, the contrast is not evidence that removing the belief caused the change. A controlled replay can hold the other inputs fixed and compare recorded decisions; without that control, the result must be labelled illustrative. Even controlled replay tests a particular recorded process, not every hidden influence of a general model. An explicit instruction disabling a use of personal state must be enforced at the external boundary where that use changes presentation, without requiring a complete interpretation of the model’s internal state. If unrecorded model influences can still hide the assumptions after Iris disables that use, the control has failed.
There is useful prior research here, but it answers a narrower question than this book. Lim, Dey and Avrahami studied the intelligibility of context-aware systems through why and why not explanations. Kulesza and colleagues studied explanatory debugging, in which people inspect and correct a learning system. Both motivate making decisions explainable and repairable; neither establishes that this personal-memory architecture preserves authority, discovers every dependent use or improves Iris’s work. Those claims require a separate experiment with this channel.
A proposed explanation-and-repair test The architecture can be tested against simpler alternatives without pretending the scripted fixture is a user study. Prepare a set of source-grounded technical tasks with deliberately planted, independently adjudicated errors: an expertise belief applied outside its scope, an obsolete preference used despite a current override, a policy adapter that hides too much, an omitted source qualification and a cached view that survives correction. Include legitimate cases where a belief is helpful, so a system that disables every inference cannot win by refusing to adapt. Freeze sources, authorisations, events and expected outcomes before evaluating any condition.
Compare four experiences: a task-appropriate generic interface with manual expansion; adaptation with a plain why explanation; adaptation with provenance and scope inspection; and adaptation with consequence-linked inspection plus bounded correction and dependency invalidation. Keep source access and basic accessibility available in every condition. Readers should complete the same tasks and attempt to locate and repair the planted mistakes; independently inspect the actual decision logs to check whether displayed explanations describe recorded causes.
Record omitted qualifications, unauthorised uses, mistaken scope extensions, time to find a defective use, successful repairs, residual downstream effects, unnecessary disclosure of personal evidence and work imposed by the interface. Test a later session to see whether a rejected belief returns through re-derivation. Separate performance on deterministic fixture checks from reader comprehension and control. No such experiment has been run for this book; a higher trust rating or a convincing explanation would not by itself count as a successful repair.
Running it Running it. npm run example:personal -- ch27 in the reference implementation (ernanhughes/project-language) prints a compact account of consequential beliefs. npm run example:personal -- all includes /personal why inf-001 and /personal correct inf-001 wrong <reason>, with a supplied reason. These are scripted, bounded examples; they do not reproduce a natural reader’s discovery and correction of the opening failure.
What it shows: the record separates proposition, origin, evidence, supported and applied scope, confidence, validity, authority and known consequences. The displayed example exposes the unsupported scope expansion: being allowed to use a belief cannot make it valid for a new algorithm. wrong rejects a belief while preserving history; forget retracts it; /personal block is a separate instruction that prevents re-derivation in the example. Correction invalidates known direct and transitive dependants; a later operation must recompute them. The full run prints the invalidated plans before constructing a fresh plan. That is a test of recorded mechanisms, not a proof that the renderer catches every material omission.
What it does not show: that readers can discover errors faster, that explanations faithfully capture every influence of a real model, that the consequence threshold is calibrated, or that all dependent uses have been found. Previously delivered or acted-on effects are not reversed. The adapter does not implement every correction proposed in this chapter: it cannot narrow scope, stop one selected use or revise standing policy through the shown commands. Retraction is not physical deletion, and forget does not itself block re-derivation. No cross-service repair or human-benefit result is demonstrated.
The limit of visibility The point is an inspectable set of consequential claims and uses. A fluent paragraph describing Iris as an expert who likes detail would discard the distinctions she needs: which part she declared, which part was observed under a temporary condition, which part was inferred, and what each part changes now.
Organising claims by use makes local correction possible without licensing a comprehensive personal inventory. Records that never qualify for a legitimate use need not be kept simply to make a future why screen richer. Equally, an audit of a suppressed item must not leak content that the reader was not authorised to see. Inspectability has its own attention, privacy and retention costs, and those costs belong in the evaluation.
The examples establish no measured improvement in repair, no calibrated consequence thresholds and no complete cross-service correction. What they establish is a specification for the next test: a consequential belief needs traceable provenance, supported and applied scope, temporal standing, permitted uses and an identifiable effect. A correction needs a recorded disposition and dependency checks. Explanations must be faithful to the decision trail, and visibility must never substitute for source validity, authorisation or enforcement.
A final tension remains. Even if every retained belief were inspectable and individually justifiable, the accumulation might be more extensive than communication requires. Explanations have costs; so do collection, correction, retention and the possibility that apparently harmless beliefs reveal more in combination. Before adding one more defensible claim, the channel must ask whether it needs the claim at all. That leads to the composition of the personal layer and the question beyond it: how much of a person should the system try to model?
References Lim, B. Y., Dey, A. K. & Avrahami, D. (2009). Why and why not explanations improve the intelligibility of context-aware intelligent systems. Proceedings of CHI 2009, pp. 2119–2128. DOI: 10.1145/1518701.1519023. (Evidence about explanation interfaces in a controlled study, not evidence for this chapter’s memory system.) Kulesza, T., Burnett, M., Wong, W.-K. & Stumpf, S. (2015). Principles of explanatory debugging to personalize interactive machine learning. Proceedings of IUI 2015, pp. 126–137. DOI: 10.1145/2678025.2701399. (Background on inspectable explanations and user correction, not a validation of the proposed repair architecture.)