Shannon Entropy Calculator

Shannon Entropy Calculator

Shannon entropy quantifies the average "surprise" or uncertainty contained in a probability distribution - a foundational concept from information theory that underpins much of modern data science and machine learning.

H = -sum(p_i x log₂(p_i))

Example

A distribution with probabilities 0.5, 0.25, 0.25:

H = -(0.5 x log2(0.5) + 0.25 x log2(0.25) + 0.25 x log2(0.25)) = 1.5 bits

Intuition

A distribution where one outcome is near-certain (like 0.99, 0.01) has very low entropy - there's little surprise in the result. A perfectly uniform distribution across many equally likely outcomes has maximum entropy, since every outcome is equally unpredictable. Entropy is measured in bits when using log base 2, representing the theoretical minimum number of yes/no questions needed on average to identify the outcome.

Where This Shows Up in Machine Learning

Entropy underlies decision tree splitting criteria (information gain measures how much a split reduces entropy), cross-entropy loss functions used to train classification models, and general measures of how predictable a dataset's class labels are before any modeling even begins.