Learn & Understand

How Machines Read: The Curious Unit Called the Token

The token count estimator gives a rough conversion from words to tokens, the units that language models actually process. A token is not quite a word and not quite a character; it is something in between, and understanding this curious unit reveals how machines read text, why tokenization matters for cost and capacity, and the clever compromise that lets models handle the vast, irregular vocabulary of human language efficiently. The token is a small but foundational idea behind modern language models.

Neither Words Nor Letters

A machine processing text must break it into pieces, and there are two obvious choices: split into whole words, or split into individual characters. Both have problems. Whole words create an enormous vocabulary, since languages have vast numbers of words, and struggle with rare or novel words never seen before. Characters keep the vocabulary tiny but lose the meaning carried by whole words, forcing the model to reassemble everything from letters. Neither extreme is ideal, and the token is the compromise between them.

The Subword Compromise

Modern models use tokens that are typically sub-word pieces: common words become single tokens, while rarer or longer words are split into several smaller tokens, chunks of characters that recur across many words. This clever scheme keeps the vocabulary manageable, since a limited set of common pieces can build up any word, while still capturing meaning by keeping common words whole. A rare word the model has never seen can still be represented by combining familiar pieces. The token strikes the balance between the vast vocabulary of words and the meaningless simplicity of letters.

Why Token Counts Matter

Tokens are the currency of language models in a very practical sense. A model can only process a limited number of tokens at once, its context window, and services are often priced by the number of tokens processed. So knowing roughly how many tokens a piece of text will become is important for planning, both to stay within limits and to estimate cost. Because tokens do not map one-to-one to words, a rough conversion factor is needed, since a given amount of text becomes somewhat more tokens than words on average.

A Rough Guide, Not an Exact Count

The calculator estimates tokens from a word count using a typical ratio, useful for quick planning. But it rightly notes that the exact count varies: different models use different tokenization schemes, other languages tokenize differently, and technical or unusual text tends to fragment into more tokens than plain prose. The estimate is a guide, not a precise count, which would require the specific tokenizer of the model in question. The calculator makes the abstract token count tangible for planning, illuminating the humble sub-word unit through which machines read all the text they process.

For how those tokens become vectors, see the Embedding Layer Size Calculator; to size a full network, the Neural Network Parameter Calculator.