What Is RLHF?

From next-token predictor to assistant

A freshly pretrained transformer (see How Transformers Work) is very good at predicting the next token in a sequence, but that's not the same thing as being a helpful assistant. Left alone, a base model will happily continue a prompt in whatever direction its training data suggests — including unhelpful, rambling, or unsafe directions. RLHF (Reinforcement Learning from Human Feedback) is the training step that turns a raw predictor into something that behaves like ChatGPT, Claude, or Gemini.

The three-stage process

1. Supervised fine-tuning (SFT) — the base model is shown thousands of example conversations written by humans (a question, followed by a good answer), and trained to imitate that style directly.
2. Reward modeling — human raters compare pairs of model outputs for the same prompt and pick which one is better. A separate "reward model" is trained on these comparisons to predict which of two responses a human would prefer.
3. Reinforcement learning — the original model is then fine-tuned to generate responses that score highly according to the reward model, using an RL algorithm (commonly a variant of PPO). It's rewarded for helpfulness and penalized for the behaviors humans marked as bad.

Why it matters

RLHF is a big part of why two models built on similar underlying architectures can feel very different to talk to — the base transformer sets the ceiling on raw capability, but RLHF (and related techniques like DPO) shapes tone, safety behavior, and how directly a model follows instructions. Providers rarely publish exact RLHF details, which is one reason models from different labs have noticeably different "personalities" even when their benchmark scores are close.