Perplexity Calculator (Language Models)

Perplexity Calculator

Perplexity is one of the standard ways to evaluate how well a language model predicts text - it can be thought of as the model's average "branching factor," or roughly how many equally likely next-word choices it's effectively confused between at each step.

Perplexity = e^{(Cross-Entropy Loss)}

Example

A model with a cross-entropy loss of 2.0 nats:

Perplexity = e^2.0 = 7.39

Interpreting the Number

Lower perplexity is better - it means the model assigns higher probability to the words that actually appear next in real text. A perplexity of 1 would represent a model that predicts the correct next token with total certainty every single time, which never happens in practice with real, ambiguous language. A perplexity of 7.39, in this example, is roughly equivalent to the model being as uncertain as if it were choosing uniformly among about 7 equally likely next words.

A Useful Caveat

Perplexity values are only directly comparable between models evaluated on the same dataset with the same tokenization scheme - a lower perplexity on a different test set or a different vocabulary doesn't necessarily mean a better model, since the difficulty of the underlying prediction task itself can vary significantly.