GPU Memory Calculator
Why a "7B Model" Needs Far More Than 7 Times 4 Bytes
Model parameter count alone dramatically understates how much GPU memory training actually consumes. Each parameter needs storage not just for its value but for its gradient and, with the widely used Adam optimizer, two additional moment estimates — roughly four times the raw parameter memory before a single activation is stored. This calculator applies that standard approximation, plus an optional activation-memory term for a given batch size.
The Formula
The default training multiplier of 4 accounts for the parameter values themselves plus gradients and Adam's two optimizer moment buffers, each stored at 4 bytes (32-bit) per parameter.
Parameter-Only Memory by Model Size
| Model size | Parameters | Estimated memory |
|---|---|---|
| 125M | 125,000,000 | 2.00 GB |
| 1B | 1,000,000,000 | 16.00 GB |
| 7B | 7,000,000,000 | 112.00 GB |
| 13B | 13,000,000,000 | 208.00 GB |
| 70B | 70,000,000,000 | 1,120.00 GB |
This is a standard rough estimate for full-precision training; using mixed precision, gradient checkpointing, or parameter-efficient fine-tuning methods can reduce actual usage well below this figure. Activation memory (which scales with batch size and sequence length) is additional and not shown in this table.
Where This Matters
- Hardware selection — deciding whether a model fits on a single GPU, needs multiple GPUs, or requires model parallelism starts with this kind of estimate.
- Batch size planning — once parameter memory is accounted for, remaining GPU memory sets the ceiling on batch size via the activation-memory term.
- Cost forecasting — memory requirements determine which cloud GPU instance type is needed, which in turn sets the hourly training cost.
How to Use This Calculator
- Enter Model Parameters (count).
- Optionally adjust the Training Multiplier (defaults to 4 for Adam).
- Optionally enter Batch Size and Activation Memory per Sample (bytes) to include activation memory in the estimate.
- Select Calculate to see the approximate total GPU memory required, in gigabytes.
Related Calculations
Once memory requirements are known, estimate the bill with the Training Cost Calculator, or check how batch size affects both memory and speed with the Batch Size Calculator.