Neural Network FLOPs Calculator

Neural Network FLOPs Calculator

FLOPs (floating point operations) is the standard way to measure and compare the computational cost of neural network layers, independent of specific hardware.

Formula

Multiply-Add Operations = Input Features x Output Features x Batch Size
FLOPs = Multiply-Add Operations x 2 (one multiply + one add per operation)

Example

A dense layer with 512 input features, 256 output features, batch size 32:

Multiply-Adds = 512 x 256 x 32 = 4,194,304 → FLOPs = 8,388,608

This Covers One Layer's Forward Pass Only

This calculation covers a single dense layer's forward pass only - estimating a full model's total FLOPs requires summing this same calculation across every layer in the network, accounting for each layer's specific input/output dimensions and architecture type (convolutional, attention, recurrent layers each have their own distinct FLOPs formulas).

Training Costs More Than Inference

Training a model requires both a forward pass and a backward pass (backpropagation) to compute gradients, which typically costs roughly 2-3x the FLOPs of inference (forward pass) alone - this is one reason training a large model is dramatically more computationally expensive than simply running inference on an already-trained model.