What Is Pairwise Comparison Data? The Specific Format Behind Preference Training
The concrete data format most preference training actually uses: exactly two responses shown side by side, with a single judgment about which one is better.
Exactly two responses, one judgment
Pairwise comparison data presents a rater with exactly two responses to the same prompt and asks for a single judgment: which one is better. It's the most common concrete format preference data actually takes, as opposed to ranking many responses at once or assigning an absolute quality score.
Why two at a time, rather than ranking a longer list
Judging which of two responses is better is a much easier and more consistent task for a human rater than ranking five or ten responses relative to each other simultaneously — pairwise comparisons tend to produce more reliable, consistent judgments precisely because they simplify the decision to a single binary choice.
How individual pairwise judgments add up to something useful
A large collection of pairwise judgments, gathered across many different pairs and prompts, gives a reward model enough signal to learn a more general, continuous sense of quality — even though each individual judgment only ever compares two responses directly against each other.
Frequently Asked Questions
What is pairwise comparison data?
Training data where a rater is shown exactly two responses to the same prompt and gives a single judgment on which one is better — the most common concrete format preference data takes.
Why use pairwise comparisons instead of ranking many responses at once?
Judging which of two responses is better is an easier, more consistent task for a human rater than ranking many responses simultaneously, which tends to produce more reliable training data.