Learn & Understand

Counting Operations: FLOPs as the Currency of Compute

The neural network FLOPs calculator counts the floating-point operations a layer performs, the basic arithmetic steps of multiplication and addition. FLOPs have become the standard currency for measuring the computational cost of models, a hardware-independent way to compare how much work different networks require. Understanding what FLOPs measure, why they are counted, and how they connect to the sweeping scaling laws of modern machine learning reveals why this humble count of arithmetic operations has become so central to the field.

The Fundamental Unit of Work

At its core, a neural network's computation is a vast number of simple arithmetic operations, chiefly multiplications and additions of numbers. A floating-point operation, or FLOP, is one such basic arithmetic step. Counting the FLOPs a model requires measures the sheer amount of arithmetic work it performs, independent of how fast any particular machine can do it. This makes FLOPs a clean, universal measure of computational cost: two models can be compared by how many operations they demand, regardless of the hardware they run on.

Why Not Just Time?

One might measure cost simply by how long a model takes to run, but time depends heavily on the specific hardware, its speed, its efficiency, its condition, so a time measured on one machine says little about another. FLOPs sidestep this by counting the intrinsic work rather than the elapsed time, giving a figure that reflects the model itself rather than the machine. This hardware independence is precisely why FLOPs are used to compare architectures and to reason about computational cost in the abstract, before any particular hardware is chosen.

Training Costs More Than Running

FLOPs also illuminate an important asymmetry: training a model costs far more computation than running it. Making a prediction requires a single forward pass through the network, but training requires that forward pass plus a backward pass to compute how to adjust the parameters, and this is repeated across enormous amounts of data many times. So the FLOPs to train a model dwarf the FLOPs to use it, which is why training is such a computationally expensive undertaking compared with everyday inference. The count makes this gap concrete.

The Currency of Scaling Laws

Perhaps most profoundly, FLOPs have become the currency in which the scaling of machine learning is measured. Research has found remarkably regular relationships between the amount of computation poured into training, measured in FLOPs, and the resulting capability of models, so that FLOPs serve as a fundamental axis along which progress is charted. The total compute of a training run, in FLOPs, has become a headline figure for the scale of an effort. The calculator counts the FLOPs of a layer, one piece of this vast accounting, quantifying the basic arithmetic work that has become the standard measure, and driver, of what machines can learn.

Turn FLOPs into a latency estimate with the Model Inference Latency Calculator, or the parameters behind them with the Neural Network Parameter Calculator.