What Is Preference Data? The Fuel Behind RLHF and DPO
The specific kind of data that alignment training actually runs on: not a single correct answer, but a judgment about which of two responses is better.
A judgment between two responses, not a single correct answer
Preference data consists of examples where two (or more) responses to the same prompt are compared, with a judgment — usually from a human rater — indicating which one is preferred, rather than providing a single fixed correct answer the way supervised fine-tuning data does.
Why comparison works better than a fixed answer for this purpose
For many tasks — "explain this well," "write a helpful reply" — there's no single objectively correct answer, but a person can usually judge fairly reliably which of two given responses is better. Preference data captures exactly that more achievable kind of judgment, rather than demanding an impossible single "correct" answer for every open-ended prompt.
Where preference data actually gets used
Preference data is the training fuel behind both training a reward model for RLHF, and behind DPO, which uses the same kind of comparison data directly, without needing a separate reward model step in between at all.
Frequently Asked Questions
What is preference data?
Training examples where two or more responses to the same prompt are compared, with a judgment indicating which one is preferred — rather than a single fixed correct answer.
Why is preference data used instead of examples with a single correct answer?
For many open-ended tasks there's no single objectively correct answer, but a person can usually judge which of two responses is better fairly reliably — preference data captures that more achievable kind of judgment.