Which Tokens Receive Loss?
Turn supervision policy into exact token positions, derive its gradient consequences, and verify causal alignment before training.
An incident dataset can contain the right answers and still optimise the wrong objective. A collator might supervise the user report, omit the assistant ending, or include a tool’s observation as if it were the assistant’s desired response. The training loop will happily differentiate the resulting scalar. Nothing in backpropagation knows which behaviour the dataset author meant to teach.
Next-Token Loss and Teacher Forcing established the numerical loss. Instruction, Completion and Conversation Datasets established the rendered sequence. The remaining bridge is a versioned decision about which positions become targets. We make that decision inspectable before performing an update.
The same conversation will be compiled under several policies. We will count targets by hand, calculate loss on one fixed small model and compare gradients. This is a CPU forward/backward experiment, not an incident-triage training result or a comparison of pretrained models.
An objective includes its selection rule
Let $x_0,\ldots,x_{T-1}$ be a token sequence and $m_j\in\{0,1\}$ indicate whether position $j$ supplies a target. With the existing one-step causal convention,
The first token has no preceding logit in this convention. We therefore exclude the initial BOS target. Our full-sequence policy includes subsequent role markers, bodies and endings. The assistant-only policy excludes role markers and selects assistant bodies plus EOS. Making these details explicit prevents two implementations from using the same policy name for different arrays.
The PyTorch cross-entropy documentation describes ignored targets and reductions. Our pinned implementation uses -100 as an ignored label and rejects an empty selected-target set before a mean reduction. The sentinel is not an ordinary vocabulary ID and does not remove the corresponding input token from context.
A worked nine-position conversation
Use the unmerged byte vocabulary from Chapter 07. The user says Hi; the assistant says OK. The sequence is:
| Position | Token | Assistant-only label |
|---|---|---|
| 0 | BOS257 | −100 |
| 1 | USER260 | −100 |
| 2 | H72 | −100 |
| 3 | i105 | −100 |
| 4 | EOS258 | −100 |
| 5 | ASSISTANT261 | −100 |
| 6 | O79 | 79 |
| 7 | K75 | 75 |
| 8 | EOS258 | 258 |
There are three selected targets. Logit position5 predicts position6, position6 predicts7, and position7 predicts8. Labels remain stored at their unshifted sequence positions. The existing loss compares logits[:, :-1] with labels[:, 1:]; the data policy must not perform another shift.
For one assistant turn, completion-only and assistant-only select the same three target IDs here. Their policy identities still differ because their meanings diverge on other conversations. A cache keyed only by the observed count would not distinguish them when that distinction becomes important.
Full-sequence supervision selects eight positions in this example, excluding BOS. The model is now rewarded for predicting the user text and template structure too. A lower resulting average cannot be compared directly with the three-target assistant average as evidence that the answers improved.
The strip shows all four policies on the same nine positions. Notice that the assistant marker at position 5 carries an ignored label while its logit still predicts position 6, and that identical counts for completion-only and assistant-only here conceal different selection rules.
Multiple turns expose a hidden difference
Extend the conversation with user Why? and assistant Yes. The four bodies contain $2+2+4+3=11$ byte tokens. Four role markers and four endings add eight positions, giving20 positions including BOS. Full-sequence loss has 19 targets.
Assistant-only loss selects OK plus its ending and Yes plus its ending: $3+4=7$ targets. Completion-only selects the last assistant suffix, four targets. A deliberately wrong prompt-only policy selects the two user bodies and their endings: $3+5=8$ targets. It is a diagnostic negative control, not an approved task objective.
On the same fixed 5,064-parameter CPU model, the measured comparison was:
| Policy | Targets | Mean NLL | Output-head gradient norm |
|---|---|---|---|
| assistant-only | 7 | 5.549151 | 0.965046 |
| completion-only | 4 | 5.442819 | 1.411811 |
| full sequence | 19 | 5.598944 | 0.698461 |
| incorrect prompt-only | 8 | 5.554770 | 1.059875 |
No optimiser step occurred between rows. The differences arise from target selection and normalisation, not from one model receiving more training. The completion-only loss happens to be smaller in this example; that is not a method-selection recommendation.
Follow the gradient, not just the ignored label
For logits $z_{t,v}$ at a prediction position, differentiating the selected cross-entropy gives
Prompt embeddings can still receive gradients through the representations used to predict an answer. Their input IDs remain attended context. In the assistant-only run, the embedding gradient norm was approximately 0.151867. It would be incorrect to describe masking user targets as removing user information from the computation or protecting its privacy.
Likewise, masking does not make a leaked reference answer harmless if that answer is present in the context. A model can condition on an ignored input token. The information boundary and the target boundary are separate parts of the experiment.
Give tool observations an explicit origin
A tool’s output may be useful context while remaining an inappropriate target for assistant imitation. We preserve a tool-origin span and exclude it under assistant-only supervision. The educational adapter places that observation in a user-control slot so it can reuse Chapter07’s alternating-message compiler; it records this limitation in a different template identity. It is not a native tool-call template or a claim about Qwen’s tool protocol.
In the test conversation, Hi/OK is followed by tool observation 42 and assistant Done. The two assistant answers contribute $3+5=8$ targets. The complete tool span has ignored labels and remains context. Full-sequence training would intentionally select a different objective, including that observation; it should not be mistaken for assistant-only training with a performance optimisation.
This restricted adapter accepts only role/content text and the supported alternating structure. Production tool traces can contain call identifiers, structured arguments and several observations. They need a representation contract that preserves those relationships instead of flattening everything into apparently ordinary user text.
Choose the denominator deliberately
For examples with selected counts $N_i$, token weighting gives
A declared sample weight $w_i$ can instead define $\sum_iw_iN_iL_i/\sum_iw_iN_i$, or an explicitly sequence-weighted alternative. The choice changes the statistical objective. Our implemented path uses the existing unweighted token mean; it does not expose an unverified general sample-weighting feature. Chapter10’s accumulation helper preserves the total-target denominator across microbatches.
Empty supervision makes the denominator zero. A missing assistant boundary, all-ignored labels or response truncation should therefore be investigated at preparation time. They must not become a NaN later blamed on the optimiser. The current policy rejects implicit clipping rather than quietly exporting an empty answer.
Packing needs an attention contract
Putting two examples in one row can save padding but allows a later example to condition on an earlier one unless attention and position handling enforce the intended isolation. An EOS boundary alone is not that guarantee. Masking the first example’s labels does not stop the second example from attending to its tokens.
The current implementation rejects packing explicitly. It reuses Chapter07’s batch construction for labels and right-padding semantics, but binds the resulting batch to its own role-span adapter and selected policy. It does not claim the original all-assistant template describes a changed full-sequence objective.
Each exported inspection contains IDs, attention, labels, the loss mask, span origins, tokenizer/template/policy identities, selected-target count and diagnostics. Validation recompiles from the intended messages and compares the complete output. Rehashing altered labels, suppressing EOS, changing tokenizer identity, duplicating serialisation or shifting labels again does not pass that comparison.
The incident candidate contains much longer sequences than the educational model’s32-position capacity. Its masks can be audited without running that model. Across 48 candidate records, a separate audit reconstructed5,634 assistant targets. This establishes position accounting, not that the old tiny model can train on these complete incidents. With supervision made explicit, the next question is whether the apparent evaluation examples are actually independent of the examples used to develop the system.
Further Reading
- Hugging Face TRL contributors, “SFT Trainer” (living documentation). Read the source. Documents assistant-only and completion-only loss with one-token shift, matching the chapter’s mask policies.
- PyTorch contributors, “CrossEntropyLoss” (v2.6 documentation). Read the source. Specifies
ignore_indexand mean reduction, grounding the denominator choice and empty-mask rejection. - Long Ouyang et al., “Training language models to follow instructions with human feedback” (2022). Read the source. Separates supervised demonstration tuning from later preference optimisation, showing why the mask definition must precede behaviour claims.