What Is a Learning Rate Schedule? Changing How Big Each Training Step Is Over Time
The learning rate rarely stays fixed through an entire training run — a schedule deliberately changes it over time, and getting that schedule right matters a lot.
Changing step size as training progresses
A learning rate schedule defines how the learning rate — the size of each parameter update in gradient descent — changes over the course of training, rather than staying fixed at one value from the very first step to the very last.
Why a fixed learning rate is usually a bad idea
A learning rate that's ideal at the very start of training (large steps for fast early progress) is often too large later on, once the model is closer to a good solution and needs finer, more careful adjustments. A schedule lets the step size shrink over time to match what each phase of training actually needs.
Common schedule shapes
A typical schedule includes a warmup period at the very start, followed by a gradual decay — often smooth (cosine-shaped) or in discrete steps — for the rest of training, tuned so the learning rate reaches a very small value right around when training is expected to finish.
Frequently Asked Questions
What is a learning rate schedule?
A defined plan for how the learning rate changes over the course of training, rather than staying fixed at one value throughout — typically starting small, ramping up, then gradually decreasing.
Why not just use a single fixed learning rate for the whole training run?
A rate that's ideal for fast early progress is often too large later in training, once the model needs finer adjustments closer to a good solution. A schedule adapts the step size to match what each phase of training needs.