How Much of You Can an AI Reproduce?

Priya opens a new architecture review. The system predicts that she will choose the expanded calculation before the summary. Its record says that she chose the calculation in the previous review. That record may be correct. The prediction has yet to be tested.

It is a small distinction, easily hidden by a fluent interface. “You chose this last time” becomes “you will choose this now”, and then “I know how you think”. Each sentence makes a different claim. The evidence does not travel between them merely because the system speaks with the same confidence.

Part III asked how little personal state good communication requires. Part IV begins with a different question:

How much of a person’s observable behaviour can an AI predict, and what would count as evidence that it can?

The title names the larger temptation. This chapter starts with the smallest defensible test: a specified behaviour, on held-out cases, beyond honest baselines. Priya’s scenes and all numbers below are constructed illustrations, not results from an experiment.

Four different claims

The claims need separate names before they can receive separate tests.

RECALL        what the person previously said or did
ESTIMATION    an inference about something presently unobserved
PREDICTION    a forecast of a specified future observable outcome
REPRODUCTION  a claim to reproduce behaviour across a stated range of situations

Recall retrieves the earlier choice. Estimation might infer which explanation Priya currently prefers when she has not declared it. Prediction commits to which option she will select before she selects it. An estimation task can also be evaluated against a later observation, but the claim remains about the present condition it estimated. Neither its name nor a successful score grants permission to infer her mood, stress or motivation.

Reproduction is a broader claim about a range of behaviours and situations. It must name that range. Matching several interface choices would test those choices; it would not establish that the system reproduces Priya across her work, relationships and decisions. A benchmark can provide evidence for bounded behavioural similarity. It cannot turn its untested remainder into a result or settle identity by calling the model a replica.

These distinctions also expose a misleading test. If an answer is already in the system’s accessible record, recovering it can demonstrate recall. It does not demonstrate prediction of a new choice. Writing an answer in Priya’s style establishes neither.

Name the outcome

“Predicts Priya” is too large to score. “Predicts which of three views she selects first in an architecture review” has an observable outcome. So does predicting whether she requests the derivation within that review. Each requires its own eligible cases, evidence cutoff and scoring rule.

The prediction also needs a context-and-horizon contract:

PREDICTION = target + context + time horizon

target:   choice of explanation depth
context:  architecture review of a familiar subject,
          with no declared time constraint or urgent objective
horizon:  the next comparable review within seven days of the forecast

“Priya will choose the terse explanation” leaves those boundaries unstated. A forecast for the next ordinary review is not a forecast for an incident six months later. Evidence about one cannot silently establish the other.

prediction error ≠ person changed ≠ context changed

These possibilities can overlap. If a forecast within its stated contract misses the observed choice, it remains a miss even when a declared revision or changed condition helps explain it. The miss does not automatically falsify an earlier account of her earlier behaviour, and a different choice alone does not establish what changed. Eligibility and scope must be declared in advance; difficult cases cannot be excluded by narrowing the contract after their outcomes are known.

The target matters because an observable action can be a poor substitute for the thing its name suggests. Not requesting more detail does not, by itself, show that Priya understood the explanation. Selecting a summary does not establish that she prefers summaries generally. She may have accepted an adequate option, followed a default or postponed the question. A test of understanding needs evidence of understanding; a choice log cannot quietly supply it.

The same restriction applies to reasons and values.

predicting a choice
        ≠ predicting a reason
        ≠ predicting a value
        ≠ reproducing a person

A system may correctly predict the table because that is the common selection. It has not thereby predicted why Priya chooses it. Predicting the reason she will report is another observable target, with its own test; a matching report still does not prove access to her inner reasoning. A claim about a value needs to specify what evidence would support it across the situations in which that value is supposed to matter.

Scope expansion remains a new inference, as in Chapter 27. Success at predicting the first view in a technical review cannot be renamed success at predicting Priya’s decisions. Even within one task, choosing a view, finding an explanation sufficient and acting on it are different outcomes. There is no earned universal ordering of their difficulty here.

Hold the future out

The essential boundary is temporal: what was available when the forecast was made, and what became known afterward?

For a prediction of Priya’s next selection, the permitted evidence and the forecast must be recorded before that selection. Later corrections, subsequent messages and descriptions written with knowledge of the outcome cannot enter as if they had been available earlier. Changing the forecast after seeing the answer produces a reconstruction of success.

Held-out cases must be genuinely new for the claim being tested. Moving a known answer into another file does not hold it out. Near-duplicate questions or summaries of the same event can carry the answer across an apparent split. If the claim concerns future interactions, the separation must respect their order, including any updates allowed between them. If it concerns unfamiliar situations, familiar variations alone do not establish it.

The record should make the target, available inputs, cutoff, forecast, outcome and scoring decision inspectable. That is a measurement obligation, not permission to collect every interaction. Evidence has to be legitimately available for this use. A better score cannot retrospectively authorise the observations, retention or inferences used to obtain it.

Compare with the plain alternatives

Correct predictions are only the beginning. Perhaps most people select the expanded calculation in this kind of review. Perhaps Priya simply repeats her last selection. A personal predictor has to show what it adds beyond those explanations.

Two comparisons belong beside its score:

• Generic default: a declared rule that does not use Priya’s individual history, such as the common choice for this task, established without the held-out answers.
• Repeat-last-choice: the last legitimately available choice in a comparable task, with the meaning of comparable fixed before seeing the new outcome.

When there is no prior comparable choice, the fallback and the number of affected cases must be visible. Quietly discarding those cases gives the personal predictor an easier population than the one it is claimed to serve.

The comparisons also need fair access to current context. If Priya has explicitly declared five minutes available, a generic rule may use that declaration too. Giving only the personal predictor today’s task and time limit would confound the contribution of personal history with the contribution of current information. A result must say which difference it actually tests.

A predictor that ties these baselines may still generate plausible answers. The tie supplies no evidence that its additional personal state improved prediction. Beating them supplies evidence for the stated target and cases, with uncertainty; it does not establish superiority everywhere else.

Compare Priya with herself, carefully

There is a third comparison: Priya’s own retest consistency. If she answers a comparable question at two times, how often do the answers agree? The interval, conditions and number of paired responses matter. Her answer last month and her answer today may differ because the task, available information or her declared objective changed.

This comparison is agreement between two responses. It is not Priya forecasting her future response. Nor is it automatically the highest score any predictor could achieve. A predictor with legitimate current information might anticipate a change that repeating her old answer would miss.

Retest agreement therefore provides context for a score, not a universal ceiling. A model matching that agreement might also be matching a trivial default. Dividing its accuracy by retest agreement cannot, by itself, establish that it has captured the person. The raw counts, baselines and conditions must remain visible.

The retest results form a reference distribution under the instrument used. Comparison is meaningful only when the target, population, timing and conditions are comparable. A predictor might exceed observed retest agreement because the repeated responses include transient mistakes or because it has relevant current information. Matching the retest number means that two measured agreements have similar values under those conditions; it does not mean that the model is as good at being Priya as Priya is.

The later choice is the outcome for a forecast of that later choice. A historically correct record can remain correct while its use as a forecast fails. Conversely, a changed answer is not evidence that the earlier record was fabricated or that Priya has become inconsistent in some durable personal sense. Chapter 23’s separation of historical truth from current applicability still applies.

Coverage is not accuracy

A predictor may decline to forecast where it has insufficient evidence. That gives a second quantity to report, rather than a way to make difficult cases disappear.

Suppose there are one hundred eligible cases. The system predicts twenty and gets nineteen right.

The last eighty cases are not eighty demonstrated successes or failures. They are uncovered cases. A headline of “95% accurate” leaves out most of the task if it does not also report twenty-percent coverage.

The selection rule must operate before the outcome is known. Baselines should be scored on the same selected cases as well as on the full eligible set; comparisons at similar coverage can expose a predictor that looks stronger only because it attempts easier cases. Results should show how accuracy changes as coverage changes, rather than present one favourable pair as a universal ranking.

Deliberate abstention also differs from an invalid response, an execution failure or an outcome that was never observed. Those cases require explicit accounting. Silently removing them changes the denominator and can improve the score without improving prediction.

Abstention may be appropriate, but it has a consequence: Priya might receive a default view, choose manually or wait for a question. Whether that is better than an occasional wrong prediction depends on the interaction and the cost of error. Accuracy and coverage alone do not measure that benefit.

Keep changed contexts visible

Return to a smaller constructed case. Over ten reviews, the personal predictor gets eight first-view choices right. The generic task default also gets eight right, and repeating Priya’s last comparable choice gets eight right. In this illustration, all three succeed on the same routine cases and miss the same two changes.

One changed case is a review before a meeting, with five minutes explicitly available. The other is an incident investigation with a declared immediate objective of locating the failure. Both call for a different first view from the routine architecture review. On these constructed counts, the personal predictor has no advantage over either baseline despite its eighty-percent score.

The result also hides a distribution: eight successes in eight routine cases, none in two changed cases. These tiny counts establish no general failure rate. They show why the counts and conditions must remain beside the aggregate. If a system abstains on both changed cases, its routine accuracy may remain perfect while its coverage there is zero.

Relevant conditions can be the current task, device, explicit mode, available time or declared objective. They need not become diagnoses of fatigue or motivation. Conditions used to select a forecast must have been available before the outcome; later analysis can identify additional groups only if it is labelled as such. Otherwise a convincing story about the misses can masquerade as a tested predictor.

Reporting by context change also guards against treating an old preference as a permanent target. If Priya revises her standing instruction, prediction should be assessed against the behaviour actually being forecast, with the revision’s time and scope visible. Success on the previous policy does not demonstrate success after it changed.

State the conditions of the prediction

The interface itself can alter the outcome. If the system predicts that Priya will choose the summary and makes it the default, her eventual selection is behaviour under that presentation. It is not evidence of what she would have selected under an unchanged interface. Whether the default helps her is a further question.

Behaviour observed under a temporary overlay carries that provenance, as Chapter 24 requires. A successful prediction during an explicitly brief review cannot silently become a standing preference for brevity. Measurement does not suspend the rule against indirect durable learning from a temporary condition.

A sequence introduces another boundary. Predicting each next choice while being given Priya’s actual preceding choices tests forecasts with observed history. Generating an entire sequence from the starting point tests behaviour when later inputs may include the model’s own earlier predictions. The two tests supply different evidence. An early mismatch can change the simulated conditions of later choices; individual-choice accuracy does not establish whole-sequence fidelity. Nor can an independence assumption about errors be slipped into a sequence score without justification.

pointwise predictive accuracy ≠ trajectory fidelity

Three correct isolated choices do not establish that the system captures how Priya moves between them. New evidence may lead her to revise a view; that revision changes her next choice, and a commitment made there may constrain the choice after it. A reproduction claim across that sequence must test those dependencies and revisions, not merely count matching endpoints. Context shifts, changed minds, uncertainty, abstention and conflicting values belong in the test wherever they fall within its claimed range.

These distinctions prevent “predicts her behaviour” from concealing which behaviour, under whose presentation, with which preceding information. A usable result names those conditions and its limits. It gives counts, uncertainty and the range of tasks, times and people tested. A result for Priya does not automatically apply to another person, just as an average across people does not show how well Priya herself was predicted.

Prediction is not communication benefit

The book’s interest remains communication. A small, scoped predictor might help choose a first view without attempting to reproduce a person. That is a possibility to test, not a result earned by a prediction score.

The communication question needs its own outcomes: whether the view preserved what mattered, supported understanding or an appropriate decision, and reduced avoidable effort under the applicable constraints. A predicted choice might be easy to make and still communicate badly. A less frequently selected view might preserve a qualification the preferred one omits.

communication benefit
        ≠ predictive accuracy
        ≠ behavioural reproduction
        ≠ identity
        ≠ authority

More personal evidence is not justified merely because it raises an accuracy score. Its legitimate use and the benefit of the resulting communication still require their own account. Part III’s question about necessity survives the move into prediction.

The defensible claim at the end of this chapter is therefore bounded: on these held-out cases, in this context and over this horizon, using this permitted evidence, the system forecast this observable outcome with this coverage and accuracy, compared with these baselines and this retest agreement. Nothing in the constructed examples establishes that result for the book’s proposed system. They specify what an eventual claim would have to show.

Now suppose a predictor eventually does very well on meeting decisions. A colleague asks Priya to meet on Tuesday afternoon, and the system forecasts acceptance. Even if the forecast proves correct, it has answered only what she would choose. Whether it may send the acceptance is a different question. Chapter 30 begins where measurement ends: a model of Priya is not Priya, and knowing her likely answer does not grant authority to give it.