← Fine Tuning From First Principles

Which Tokens Receive Loss?

Turn supervision policy into exact token positions, derive its gradient consequences, and verify causal alignment before training.

An incident dataset can contain the right answers and still optimise the wrong objective. A collator might supervise the user report, omit the assistant ending, or include a tool’s observation as if it were the assistant’s desired response. The training loop will happily differentiate the resulting scalar. Nothing in backpropagation knows which behaviour the dataset author meant to teach.

Next-Token Loss and Teacher Forcing established the numerical loss. Instruction, Completion and Conversation Datasets established the rendered sequence. The remaining bridge is a versioned decision about which positions become targets. We make that decision inspectable before performing an update.

The same conversation will be compiled under several policies. We will count targets by hand, calculate loss on one fixed small model and compare gradients. This is a CPU forward/backward experiment, not an incident-triage training result or a comparison of pretrained models.

An objective includes its selection rule

Let $x_0,\ldots,x_{T-1}$ be a token sequence and $m_j\in\{0,1\}$ indicate whether position $j$ supplies a target. With the existing one-step causal convention,

$$ L(\theta;m)=\frac{1}{N}\sum_{j=1}^{T-1}m_j \left[-\log p_\theta(x_j\mid x_{ Changing $m$ changes both the terms and their denominator. A full-sequence language-modelling objective asks the model to predict the prompt as well as the answer. An assistant-only objective asks it to predict selected responses given their context. A completion-only objective selects a declared suffix. These objectives can share input IDs and produce different gradients.

The first token has no preceding logit in this convention. We therefore exclude the initial BOS target. Our full-sequence policy includes subsequent role markers, bodies and endings. The assistant-only policy excludes role markers and selects assistant bodies plus EOS. Making these details explicit prevents two implementations from using the same policy name for different arrays.

The PyTorch cross-entropy documentation describes ignored targets and reductions. Our pinned implementation uses -100 as an ignored label and rejects an empty selected-target set before a mean reduction. The sentinel is not an ordinary vocabulary ID and does not remove the corresponding input token from context.

A worked nine-position conversation

Use the unmerged byte vocabulary from Chapter 07. The user says Hi; the assistant says OK. The sequence is:

Position Token Assistant-only label
0 BOS257 −100
1 USER260 −100
2 H72 −100
3 i105 −100
4 EOS258 −100
5 ASSISTANT261 −100
6 O79 79
7 K75 75
8 EOS258 258

There are three selected targets. Logit position5 predicts position6, position6 predicts7, and position7 predicts8. Labels remain stored at their unshifted sequence positions. The existing loss compares logits[:, :-1] with labels[:, 1:]; the data policy must not perform another shift.

For one assistant turn, completion-only and assistant-only select the same three target IDs here. Their policy identities still differ because their meanings diverge on other conversations. A cache keyed only by the observed count would not distinguish them when that distinction becomes important.

Full-sequence supervision selects eight positions in this example, excluding BOS. The model is now rewarded for predicting the user text and template structure too. A lower resulting average cannot be compared directly with the three-target assistant average as evidence that the answers improved.

Supervision masks over one nine-position conversation

The strip shows all four policies on the same nine positions. Notice that the assistant marker at position 5 carries an ignored label while its logit still predicts position 6, and that identical counts for completion-only and assistant-only here conceal different selection rules.

Multiple turns expose a hidden difference

Extend the conversation with user Why? and assistant Yes. The four bodies contain $2+2+4+3=11$ byte tokens. Four role markers and four endings add eight positions, giving20 positions including BOS. Full-sequence loss has 19 targets.

Assistant-only loss selects OK plus its ending and Yes plus its ending: $3+4=7$ targets. Completion-only selects the last assistant suffix, four targets. A deliberately wrong prompt-only policy selects the two user bodies and their endings: $3+5=8$ targets. It is a diagnostic negative control, not an approved task objective.

On the same fixed 5,064-parameter CPU model, the measured comparison was:

Policy Targets Mean NLL Output-head gradient norm
assistant-only 7 5.549151 0.965046
completion-only 4 5.442819 1.411811
full sequence 19 5.598944 0.698461
incorrect prompt-only 8 5.554770 1.059875

No optimiser step occurred between rows. The differences arise from target selection and normalisation, not from one model receiving more training. The completion-only loss happens to be smaller in this example; that is not a method-selection recommendation.

Follow the gradient, not just the ignored label

For logits $z_{t,v}$ at a prediction position, differentiating the selected cross-entropy gives

$$ \frac{\partial L}{\partial z_{t,v}} =\frac{m_{t+1}}{N}\left(p_{t,v}-\mathbf1[v=x_{t+1}]\right). $$
If the next target is ignored, the direct logit gradient at that prediction position is zero. The experiment verifies exactly zero at those ignored logit positions under every policy. Notice the shifted index $m_{t+1}$: the logit at the assistant marker can have a gradient even though the marker's own label is ignored.

Prompt embeddings can still receive gradients through the representations used to predict an answer. Their input IDs remain attended context. In the assistant-only run, the embedding gradient norm was approximately 0.151867. It would be incorrect to describe masking user targets as removing user information from the computation or protecting its privacy.

Likewise, masking does not make a leaked reference answer harmless if that answer is present in the context. A model can condition on an ignored input token. The information boundary and the target boundary are separate parts of the experiment.

Give tool observations an explicit origin

A tool’s output may be useful context while remaining an inappropriate target for assistant imitation. We preserve a tool-origin span and exclude it under assistant-only supervision. The educational adapter places that observation in a user-control slot so it can reuse Chapter07’s alternating-message compiler; it records this limitation in a different template identity. It is not a native tool-call template or a claim about Qwen’s tool protocol.

In the test conversation, Hi/OK is followed by tool observation 42 and assistant Done. The two assistant answers contribute $3+5=8$ targets. The complete tool span has ignored labels and remains context. Full-sequence training would intentionally select a different objective, including that observation; it should not be mistaken for assistant-only training with a performance optimisation.

This restricted adapter accepts only role/content text and the supported alternating structure. Production tool traces can contain call identifiers, structured arguments and several observations. They need a representation contract that preserves those relationships instead of flattening everything into apparently ordinary user text.

Choose the denominator deliberately

For examples with selected counts $N_i$, token weighting gives

$$ L_{\mathrm{token}}=\frac{\sum_iN_iL_i}{\sum_iN_i}. $$
Sequence weighting gives $L_{\mathrm{sequence}}=n^{-1}\sum_iL_i$. These are equal only under appropriate equal-count conditions. A long response contributes more terms under token weighting. Equal sequence weighting gives each example equal total weight, which increases the relative contribution of each short-answer token.

A declared sample weight $w_i$ can instead define $\sum_iw_iN_iL_i/\sum_iw_iN_i$, or an explicitly sequence-weighted alternative. The choice changes the statistical objective. Our implemented path uses the existing unweighted token mean; it does not expose an unverified general sample-weighting feature. Chapter10’s accumulation helper preserves the total-target denominator across microbatches.

Empty supervision makes the denominator zero. A missing assistant boundary, all-ignored labels or response truncation should therefore be investigated at preparation time. They must not become a NaN later blamed on the optimiser. The current policy rejects implicit clipping rather than quietly exporting an empty answer.

Packing needs an attention contract

Putting two examples in one row can save padding but allows a later example to condition on an earlier one unless attention and position handling enforce the intended isolation. An EOS boundary alone is not that guarantee. Masking the first example’s labels does not stop the second example from attending to its tokens.

The current implementation rejects packing explicitly. It reuses Chapter07’s batch construction for labels and right-padding semantics, but binds the resulting batch to its own role-span adapter and selected policy. It does not claim the original all-assistant template describes a changed full-sequence objective.

Each exported inspection contains IDs, attention, labels, the loss mask, span origins, tokenizer/template/policy identities, selected-target count and diagnostics. Validation recompiles from the intended messages and compares the complete output. Rehashing altered labels, suppressing EOS, changing tokenizer identity, duplicating serialisation or shifting labels again does not pass that comparison.

The incident candidate contains much longer sequences than the educational model’s32-position capacity. Its masks can be audited without running that model. Across 48 candidate records, a separate audit reconstructed5,634 assistant targets. This establishes position accounting, not that the old tiny model can train on these complete incidents. With supervision made explicit, the next question is whether the apparent evaluation examples are actually independent of the examples used to develop the system.

Further Reading

  1. Hugging Face TRL contributors, “SFT Trainer” (living documentation). Read the source. Documents assistant-only and completion-only loss with one-token shift, matching the chapter’s mask policies.
  2. PyTorch contributors, “CrossEntropyLoss” (v2.6 documentation). Read the source. Specifies ignore_index and mean reduction, grounding the denominator choice and empty-mask rejection.
  3. Long Ouyang et al., “Training language models to follow instructions with human feedback” (2022). Read the source. Separates supervised demonstration tuning from later preference optimisation, showing why the mask definition must precede behaviour claims.