What Is Rejection Sampling Fine-Tuning? Filtering Before Training
A technique that generates many candidate responses, keeps only the best ones, and fine-tunes on that filtered set — a simpler alternative to full reinforcement learning.
Generate many, keep the best, train on those
Rejection sampling fine-tuning generates multiple candidate responses to each prompt, scores them (often with a reward model), keeps only the highest-scoring ones, and then fine-tunes the model directly on that filtered, high-quality set — using standard supervised fine-tuning on curated output, rather than a full reinforcement learning loop.
Why this is simpler than full reinforcement learning
Algorithms like PPO require carefully balancing reward optimization against training stability, which adds real complexity. Rejection sampling sidesteps most of that complexity by turning the problem back into standard supervised learning — just on data that's been pre-filtered for quality first.
The limitation compared to full RL
Because it only selects among responses the model can already generate on its own, rejection sampling fine-tuning can't push a model to discover better responses than what it was already capable of producing occasionally — full reinforcement learning can, in principle, guide a model toward genuinely new behavior beyond its existing output distribution.
Frequently Asked Questions
What is rejection sampling fine-tuning?
A technique that generates multiple candidate responses per prompt, keeps only the highest-scoring ones, and fine-tunes the model on that filtered, high-quality set using standard supervised fine-tuning rather than full reinforcement learning.
What is the limitation of rejection sampling fine-tuning compared to full RL?
It can only select among responses the model already generates on its own, so it can't push the model toward genuinely new behavior beyond what it was already occasionally capable of producing.