A Model of Me Is Not Me

A colleague asks Priya to meet on Tuesday afternoon. The system forecasts that she will accept and prepares a reply. In this constructed case, its forecast is right and its wording matches what she would have written. It has not been authorised to send the message.

That last fact does not become less important because the first two are true. The system could predict this particular choice very well and still have no standing to make it for her. “It knows what I would say” and “it may say it for me” require different evidence.

Chapter 29 established how a bounded prediction would have to be measured. Grant it success here. The question now is what that success establishes about the system’s relation to the person.

predicts me
        ≠ represents me
        ≠ is me
        ≠ is authorised to act as me

The distinctions are not stages through which enough accuracy eventually carries a model. Each asks something different: about a forecast, a representation, identity or a grant of authority.

A model may resemble me without being me, and may predict me without being authorised to act for me.

A person, a model and an actor

Priya is the person whose behaviour the system is trying to anticipate. The model is a constructed representation, with particular evidence, a history of updates and a limited range of uses. The acting system can prepare or perform operations that affect her and other people. When it has a valid grant for an operation, it can act as her proxy within that grant.

Those roles can be implemented in one product without becoming one thing. A model can supply a suggestion to an actor. The actor can then use a service to send a message or make a booking. Neither the model’s suggestion nor the service’s ability to execute supplies the missing authority.

An actor can also act without permission. Calling it an agent is not evidence that it is an authorised proxy. The unauthorised send remains an action with effects, even if every component describes itself as assisting Priya.

The separation matters to explanation. If a reply was sent, the system should be able to distinguish the prediction that informed its content from the permission under which it was sent. “The model was confident” reports confidence in a forecast. It does not answer who authorised the act.

Two meanings of representation

The word represents can hide the same movement. A model represents Priya descriptively when it contains claims about her or predicts a bounded part of her behaviour. A proxy represents her in an activity when it has standing to speak or act on her behalf. The first meaning does not confer the second.

An accurate record that Priya accepted the previous meeting is a representation of an event. A prediction that she will accept this one is a claim about a future event. A message accepting it on her behalf is an act. Their content may agree while their provenance, use and authority differ.

Descriptive representation also remains scoped. A model of Priya’s architecture-review choices does not become a representation of every judgement she makes. Its supported scope and its applied scope need to remain visible, as Chapter 27 required. Expanding from “predicts this choice” to “speaks for Priya” introduces more than another prediction target: it introduces a claim to authority that prediction cannot supply.

A system may therefore hold a true belief that she would accept and still be prohibited from using that belief to send an acceptance. She need not falsify its record to stop the act. “Yes, that is what I would choose. No, you may not choose it for me” is a coherent instruction.

Resemblance does not establish identity

Consider a constructed fork. Two models begin with identical evidence about Priya and the same results on the bounded prediction task. One is then used in her consensus work; the other is used in database-performance discussions. They receive different corrections, encounter different situations and accumulate different histories. Suppose their later suggestions diverge.

Which one is Priya? Predictive resemblance cannot answer that question. Nothing in their common evidence or equal predictive fidelity establishes either model as Priya. Their later disagreement does not tell us that Priya herself made two incompatible decisions. It tells us that two constructed models, now under different conditions, produced different outputs.

The narrow point is about what the evidence can establish. Resemblance supplies no criterion for turning either model’s continuation into the person’s own continuation. A rule based only on the shared prediction score would have treated both alike at the fork; that score cannot decide whose later answer is hers.

predictive equivalence ≠ evidence of personal identity

A personal model can approximate behaviour without acquiring identity.

This is an argument about the insufficiency of predictive equivalence, not a solution to personal identity. It does not need to decide which general theory of identity is correct. For the behavioural models considered here, the engineering consequence is plain: the model’s records, updates and outputs must remain distinguishable from Priya’s statements and decisions.

Copying the model copies a representation. It does not copy a grant of authority merely because the representation was built from her. If Priya authorises one acting system for one task, another copy’s resemblance does not make it an authorised recipient of that grant. Any extension needs its own basis in what she allowed.

The title names this boundary. The system has a model of Priya. A successful forecast does not establish that the model is Priya, and cannot be used as the missing justification for treating its output as her act.

Authority comes from the grant

Take two hypothetical systems with different performance on the same bounded prediction task. Both can prepare a flight booking.

A’s better forecast supplies no permission to purchase. B has a basis for acting that A lacks, but only within the grant’s scope and conditions. Nothing in this example establishes that B should proceed regardless of competence or uncertainty. It establishes where its authority must come from.

Priya may consider performance when deciding whether to give a grant. She may require a confirmation or sufficient evidence before an authorised operation proceeds. Those conditions matter because she imposed or accepted them. A performance score cannot manufacture a grant, waive a condition or enlarge its scope.

The proposed boundary is therefore not “a grant exists, so execute”. The grant must be valid for this actor and operation, current, and satisfied in its constraints. Other applicable restrictions still hold. Permission to book does not automatically authorise every use of personal information, and a permitted action may still need to stop when its conditions cannot be established. The later delegation chapter gives those checks their own mechanism.

Authority comes from the grant, not from resemblance.

A declared objective is also insufficient by itself. “I need to get to Berlin” can inform a proposal. It does not silently permit a purchase, a disclosure or a message in Priya’s name. The system must not turn its interpretation of the goal into permission for every operation that could help achieve it.

Correctness and permission fail separately

Suppose A purchases exactly the flight Priya would have selected, without a grant. The accurate prediction does not cure the unauthorised purchase. A later expression of satisfaction cannot be used to claim that permission existed at the time of execution.

Now suppose B purchases a flight that satisfies every condition in its valid grant, but Priya would have preferred another allowed departure. That may be poor service or a failed preference prediction. It does not become an authority violation solely because another permitted option would have suited her better.

There is a third case that must not be hidden in the second. If the grant permits only refundable flights and B buys a non-refundable ticket, B has exceeded its authority. Calling the choice a prediction error would obscure the violated constraint. Having a grant for some bookings is not permission for this booking.

correct prediction ≠ authorised action
authorised action ≠ good prediction
grant for the task ≠ permission for every proposed operation

The remedies differ. A better predictor may address the choice among permitted departures. It cannot repair the absence of permission or make an out-of-scope purchase permissible. Revising a grant may change future authority; it does not rewrite what was allowed when the earlier act occurred.

Keeping these accounts separate also preserves the person’s control. A correction to the model is not automatically a revision to a grant, just as changing a grant need not declare the model’s belief false. Priya may revoke booking authority while continuing to agree with its recommendations.

Authority has a present tense

Return to the Tuesday flight. The system prepares a purchase at 09:59. Priya withdraws its booking authority at 10:00. At 10:05 the forecast still says that she would choose the same flight. The prepared purchase must not proceed on the withdrawn authority.

Neither an old permission nor a current prediction of desire supplies a new grant. If the system believes the withdrawal conflicts with her objective, it may surface that conflict within its permitted role. It may not restore permission by deciding that the earlier instruction expressed her more authentic wishes.

Withdrawal also needs a truthful account of effects. As Chapter 26 established, removing the central record is not the same as updating every enforcement point. A queued operation and a downstream service may still hold an earlier state. The system must distinguish authority withdrawn, pending action stopped and external effect confirmed absent; an unconfirmed stop cannot be reported as completed revocation.

An operation already completed before withdrawal poses another question. Revoking future booking authority does not by itself cancel an existing reservation or undo a message already delivered. Cancellation is an operation with its own conditions and permission. The current authority record and the history of effects must not be collapsed.

These are requirements for the proposed channel, not a report that it already enforces them. Their point is to prevent a confident model from becoming the final judge of whether its permission still applies.

Whose words, whose decision, whose act?

The same one-word reply can have different origins.

"Yes."
predicted wording: the model forecasts that this is what Priya would say

"Yes."
personal adoption: the model drafted it; Priya chose and sent it

"Yes."
authorised transmission: a proxy selected and sent it under a current grant

The text alone does not identify which happened. In the first case the system is forecasting wording, with no external reply implied. In the second, Priya has adopted the statement, even if the system helped compose it. In the third, the system has performed an authorised operation on her behalf; the grant must cover that operation and its effect.

matching wording ≠ personal adoption ≠ authorised transmission

Priya need not write every word herself for a message to become a statement she adopts. But a forecast that she would adopt it cannot substitute for that adoption. Nor does approval of one draft authorise a different commitment or every later reply in the conversation.

The account should distinguish who composed the wording, who selected or approved the commitment, and what performed the transmission. A service sending a message after Priya’s explicit approval is different from a proxy deciding and sending within delegated bounds. Generated wording does not settle either case.

Attribution must follow that history. Where the system acts as a proxy, its account must not invent personal approval that never occurred. Calling the message “what Priya would have said” describes a prediction about content, not its actual origin. The later chapter on speech will ask what kind of act the reply performs; this chapter establishes why resemblance cannot supply its authority.

What the model may inform

A model can inform a proposal or help select among already permitted options when its use is allowed. It may suggest a reply, identify a possible conflict or present alternatives. Priya’s declared policies can constrain those proposals. Neither fluency nor the label proposal exempts the process from the existing rules about access, privacy and consequential use.

The distinction concerns authority, not a requirement to ask about every harmless adjustment. An existing, scoped instruction may already permit the operation. What the system cannot do is use a prediction as the missing grant or silently extend an instruction to another kind of act.

This also limits the role of temporary adaptation. An overlay may activate an authorised rule within its scope, as Chapter 25 established. It may not turn a prediction that Priya would welcome help into permission to send, purchase or commit. Her lack of correction afterward does not supply that permission retroactively.

A model can predict Priya’s reply without that prediction establishing identity, and its accuracy gives it no standing to send the reply. But there is still a question before standing can be formalised. Even if the system could be authorised to handle the meeting, which parts does Priya actually want it to handle? Should it notice the request, prepare a reply, recommend a decision, make the decision, or send the answer? Prediction does not settle that, and permission would not settle it either. Before asking what the system may do, we have to ask what Priya wants to keep.