Gradient Descent Step Calculator
Watching a Single Weight Update Happen
Every weight in a neural network moves according to the same basic rule during training: step in the direction that reduces the loss, scaled by the learning rate. This calculator performs that single update explicitly — either the vanilla version or the momentum-augmented version — so the mechanics behind an optimizer step are visible rather than hidden inside a training loop.
The Formulas
Vanilla Gradient Descent: w_new = w − learning_rate × gradient
With Momentum:
v = momentum × v_prev − learning_rate × gradient
w_new = w + v
With Momentum:
v = momentum × v_prev − learning_rate × gradient
w_new = w + v
Worked Examples
| Method | Inputs | Calculation | New weight |
|---|---|---|---|
| Vanilla | w = 0.5, gradient = 0.2, lr = 0.1 | 0.5 − 0.1 × 0.2 | 0.48 |
| Momentum | w = 0.5, gradient = 0.2, lr = 0.1, momentum = 0.9, v_prev = −0.02 | v = 0.9 × (−0.02) − 0.1 × 0.2 = −0.038; w + v | 0.462 |
Momentum carries part of the previous update's velocity forward, which is why the momentum result differs from the vanilla update even with identical weight, gradient, and learning rate.
Where This Matters
- Debugging training instability — manually verifying one step against expected output is a fast way to confirm a custom training loop is implemented correctly.
- Teaching and learning optimization — seeing the arithmetic behind a single step demystifies what "the optimizer" is actually doing under the hood.
- Comparing optimizer behavior — running the same inputs with and without momentum shows directly how momentum can accelerate or dampen a step.
How to Use This Calculator
- Enter the Current Weight.
- Enter the Gradient at that weight.
- Enter the Learning Rate.
- Optionally enter Momentum and Previous Velocity to use the momentum form (both default to 0, which reduces to vanilla gradient descent).
- Select Calculate to see the updated weight.
Related Calculations
Pick the learning rate driving this step with the Learning Rate Calculator, or estimate how long a full training run of these steps will take with the Epoch Time Calculator.