Learn & Understand

Meaning as Geometry: How Machines Turn Words Into Numbers

The embedding layer calculator counts the parameters in an embedding layer, which stores one learned vector for every item in a vocabulary. Embeddings are how machines represent words, and other discrete things, as points in a space of numbers, and the remarkable result is that meaning itself becomes geometry: similar things end up near each other, and relationships become directions. Understanding word embeddings reveals one of the most elegant ideas in machine learning, and why representing meaning numerically transformed how machines process language.

The Problem of Words as Symbols

A word is a symbol, and to a computer, symbols are just arbitrary labels with no inherent relationship to one another. Nothing about the symbols for two related words tells a machine that they are related. To process language meaningfully, a machine needs a representation in which relationships between words are somehow captured, so that it can tell which words are similar and how they relate. Simply assigning each word an arbitrary label does not do this; a richer representation is needed, one that encodes meaning.

Words as Points in Space

The solution is the embedding: each word is represented as a vector, a list of numbers, which places it as a point in a high-dimensional space. Crucially, these vectors are learned, adjusted during training so that words used in similar ways end up as nearby points. In this space, meaning becomes location: words with similar meanings cluster together, and unrelated words sit far apart. A machine can then judge similarity by measuring distance, turning the vague notion of related meaning into concrete geometry.

Relationships as Directions

The most striking property of embeddings is that relationships between words correspond to directions in the space. The direction that separates a word from its counterpart of the opposite gender, for instance, tends to be consistent across many such pairs, so that relationships can be traversed by moving along these directions. This means the geometry captures not just similarity but analogy and structure, an astonishing emergent property of learning to place words so that their usage is reflected in their positions. Meaning acquires a rich spatial structure.

The Cost of a Large Vocabulary

The calculator counts an embedding layer's parameters, which is simply one vector per vocabulary item, so the total grows with both the vocabulary size and the dimension of each vector. For large vocabularies, this can consume an enormous share of a model's parameters, which is exactly why modern language models often break text into smaller sub-word pieces to keep the vocabulary manageable. The calculator makes this cost concrete. Behind that parameter count lies a profound idea: that meaning can be encoded as position in a space, turning the elusive relationships between words into geometry a machine can measure and manipulate.

For how text is broken into vocabulary items, see the Token Count Estimator; to size the full network, the Neural Network Parameter Calculator.