What Is a Reward Model? The Judge Behind RLHF
RLHF needs a way to score responses at scale — that's the reward model's job, trained specifically to predict what a human would prefer.
A model trained to judge, not to answer
A reward model is a separate model trained specifically to score how good a response is — not to generate responses itself. It's trained on human preference data: pairs of responses to the same prompt, with a human indicating which one they preferred, until the reward model can reliably predict that preference on new, unseen responses.
Why you need a separate model for this at all
Having a human personally rate every single response generated during reinforcement learning training would be far too slow and expensive — training involves generating and scoring an enormous number of responses. A reward model automates the scoring step at scale, standing in for a human judge after being trained to approximate one.
Its role in RLHF
During RLHF, the model being trained generates a response, the reward model scores it, and that score drives the reinforcement learning update — nudging the model toward responses the reward model predicts a human would rate highly.
Frequently Asked Questions
What is a reward model?
A model trained specifically to score how good a response is, based on human preference data — used to automate response scoring at scale during RLHF training.
Why not just use human raters for every response during training?
Training involves generating and scoring far too many responses for humans to rate individually. A reward model is trained to approximate human judgment so scoring can happen automatically at scale.