What Is DPO (Direct Preference Optimization)?

The problem with classic RLHF

As covered in What Is RLHF?, the classic recipe for aligning a model with human preferences involves training a separate reward model, then using reinforcement learning (typically PPO) to optimize the original model against it. That pipeline works, but it's notoriously fiddly — reinforcement learning training is unstable, sensitive to hyperparameters, and requires keeping multiple models (the policy, the reward model, often a reference model too) in memory and in sync during training.

The DPO shortcut

Direct Preference Optimization, introduced in 2023, showed that you can skip the separate reward model and the reinforcement learning loop entirely. Given the same kind of data RLHF uses — pairs of responses where a human marked one as better than the other — DPO derives a direct mathematical relationship between the preference data and a simple loss function, and optimizes the language model against that loss directly, using ordinary supervised-learning-style training.

Why it caught on quickly

DPO is simpler to implement, more stable to train, and cheaper computationally than full RLHF, while achieving comparable results on many tasks. That made preference-based alignment practical for far more teams than the handful of labs with the infrastructure and expertise to run stable RL training pipelines — which is a big part of why open-weight model releases improved so quickly on "feels helpful to talk to" once DPO and similar simplified methods became widely known and implemented.

Not a total replacement

DPO and RLHF aren't mutually exclusive, and some frontier providers use both at different stages, or hybrid approaches inspired by each. The broader point holds either way: turning a raw pretrained model into a well-behaved assistant requires this preference-alignment step in some form, and the field has moved toward faster, simpler ways of doing it since RLHF was first popularized.