What Does a Preference Know About the Future?
We Trained a Model on Editorial Choices to See Whether It Learned What Happened Next
Most preference-learning systems use a choice to change the future.
A model produces two responses. A human selects one. The chosen response becomes positive evidence, the rejected response becomes negative evidence, and training makes outputs resembling the chosen response more likely.
The preference acts as an instruction:
Produce more things like this.
I wanted to know whether the same choice could also function as evidence.