Learn & Understand

Descending a Mountain in Fog: The Learning Rate

The learning rate calculator computes how a model's learning rate shrinks over the course of training, following one of several decay schedules. To understand why this rate matters so much, it helps to picture what training a model actually is: a blindfolded descent down a mountainous landscape, feeling for the way downhill one step at a time. The learning rate is the size of each step, and choosing it well, and shrinking it wisely, is one of the central arts of machine learning. Understanding this metaphor illuminates why schedules exist at all.

Learning as Descending a Landscape

Training a model can be pictured as descending a vast, hilly landscape in search of its lowest point, where the model performs best. But the traveler is effectively blindfolded, able only to feel the slope beneath their feet and step downhill accordingly. This is gradient descent: repeatedly taking a step in the downhill direction, guided only by the local slope, gradually working toward the valley. The whole of training is this patient, blind descent, one step after another, toward a good solution.

The Step That Can Be Too Big or Too Small

The learning rate is the length of each step. If the steps are too large, the traveler bounds recklessly across the landscape, overshooting the valley and bouncing around without ever settling into the lowest point. If the steps are too small, progress is agonizingly slow, and the descent may crawl or get stuck. A good step size is large enough to make real progress but small enough to home in on the bottom. Getting this balance right is crucial, and it is why the learning rate is such a consequential setting.

Big Strides First, Small Steps Later

The insight behind learning rate schedules is that the ideal step size changes over the course of the descent. Early on, far from the valley, large strides make sense to cover ground quickly. Later, closing in on the lowest point, small careful steps are needed to settle precisely into the bottom without overshooting. So the wise approach is to start with a larger learning rate and shrink it as training progresses, taking bold strides first and delicate steps near the end. This is exactly what a decay schedule does.

Different Ways to Slow Down

The calculator computes several schedules that differ in how they shrink the rate, some dropping in sudden steps, others tapering smoothly, others following a graceful curve that decelerates near the beginning and end. Each embodies a different strategy for the descent, and different problems suit different curves. The calculator reveals the learning rate at any point in training under each schedule, making the abstract idea of a shrinking step size concrete. Behind these curves lies a simple, vivid picture: a blind traveler descending a landscape, striding boldly at first and stepping ever more carefully as the valley draws near.

See a single step's effect with the Gradient Descent Step Calculator, or the gentle ramp-up before decay with the Learning Rate Warmup Calculator.