What Is Gradient Descent? How Models Actually Learn
The algorithm underneath every training run — a simple idea (adjust in the direction that reduces error) repeated billions of times.
The core idea
Gradient descent is the algorithm that adjusts a model's parameters during training. After each batch of examples, it measures how wrong the model's output was (the loss), calculates which direction to nudge each parameter to reduce that error, and moves every parameter a small step in that direction. Repeated billions of times, this is what "training" mechanically is.
Why small steps
The step size (called the learning rate) has to be tuned carefully — too large, and the adjustments overshoot and the training becomes unstable; too small, and training takes an impractically long time to converge. Finding a good learning rate schedule is one of the standard tuning problems in any training run.
Why it works despite the scale
It's genuinely remarkable that nudging billions of numbers a tiny amount, over and over, based only on local error signals, converges toward a model that can hold a coherent conversation. There's no global plan — just repeated small corrections in the direction of "slightly less wrong," at a scale large enough for real capability to emerge.
Frequently Asked Questions
What is gradient descent?
The algorithm that adjusts a model's parameters during training — measuring how wrong the current output is, then nudging every parameter a small step in the direction that reduces that error, repeated over and over.
What happens if the learning rate is set too high?
Training becomes unstable — parameter adjustments overshoot the correction needed and the model fails to converge properly, instead of gradually improving.