Learning Rate Decay Calculator

Learning Rate Decay Calculator

Learning rate decay gradually shrinks the step size a training algorithm takes as training progresses - big steps early help a model converge quickly, while smaller steps later help it settle precisely into a good solution instead of oscillating around it.

Decayed Rate = Initial Rate / (1 + Decay Rate x Epoch Number)

Example

Initial learning rate 0.1, decay rate 0.01, at epoch 10:

Decayed Rate = 0.1 / (1 + 0.01 x 10) = 0.1 / 1.1 = 0.0909

Why Not Just Use a Fixed Rate?

A learning rate that's too high throughout training can cause the model to overshoot good solutions and oscillate without ever converging cleanly; one that's too low throughout wastes enormous amounts of training time. Decay schedules solve this by front-loading large, fast-converging steps and progressively transitioning to small, precise refinement steps - this particular inverse-time schedule is one of several standard approaches, alongside step decay (dropping the rate at fixed intervals) and exponential decay (a smooth continuous decrease).