Learning Rate Warmup Calculator

Learning Rate Warmup Calculator

Learning rate warmup gradually ramps up from a small value to the full target rate over the first several training steps - this calculator shows the current rate at any point in that schedule.

Current LR = Target LR x (Current Step / Warmup Steps), capped at Target LR

Example

Target rate 0.001, at step 500 of a 1,000-step warmup:

Current LR = 0.001 x (500/1000) = 0.0005 (still in warmup phase)

Why Warmup Helps

Model weights start out randomly initialized, and large weight updates at full learning rate during this early period can cause training instability or even outright divergence before the model has found a reasonable starting region of the loss landscape. Warmup avoids this by starting with tiny, gentle updates and only reaching full learning rate once the model has stabilized somewhat.

Especially Important for Transformers

Warmup is especially common - and often considered essential - when training transformer-based architectures, which are notably more prone to early training instability than some other architecture types, making a proper warmup schedule a standard, largely non-negotiable part of most modern transformer training recipes.