Learn & Understand

How Tokenization Works: Why Language Models Don't See Words

The companion calculator estimates how many tokens a piece of text will use, noting that models process text as sub-word tokens rather than words, so the exact count depends on the tokenizer. Tokenization is a foundational and often-misunderstood part of how language models work, sitting between the raw text we write and the numbers a model actually processes. Understanding what tokens are, how subword tokenization strikes a balance between characters and words, and why it treats code and non-English text differently turns a token estimate into an appreciation of the hidden layer that shapes everything a language model does.

Between Characters and Words

A language model does not read text as whole words, nor as individual letters, but as tokens, chunks that are typically pieces of words, striking a balance between the two extremes.

Breaking text into whole words would require an impossibly large vocabulary and would stumble on new or rare words never seen before, while breaking it into individual characters would make sequences extremely long and give each unit little meaning. Subword tokenization is the compromise: common words become single tokens, while rarer words are split into smaller meaningful pieces, so the model has a manageable vocabulary that can still represent any text by combining pieces. A frequent word like "the" is one token, while an unusual word might break into two or three subword tokens. This is why the calculator notes that a common word may be one token while a rare word or code splits into several. Understanding that tokens are subword units, a deliberate middle ground between characters and words, is the key to how models handle language efficiently while still being able to represent anything, including words they have never seen, by assembling them from familiar pieces.

Learning the Pieces From Data

The specific set of tokens a model uses is not designed by hand but learned from a large body of text, using an algorithm that identifies which chunks of text are common enough to deserve their own token. The process starts from small pieces and repeatedly merges the most frequently occurring combinations into larger tokens, so that common sequences, frequent words, common word-endings, familiar fragments, become single tokens, while rare sequences remain split into smaller pieces. The result is a vocabulary optimized for the text the model was trained on, where the most common patterns are represented most efficiently. This is why tokenization reflects the statistics of the training data: what is common becomes a single token, what is rare gets fragmented. Different models use different tokenizers built this way, which is why, as the calculator notes, there is no single universal token count for a piece of text, it depends on the specific tokenizer. Understanding that the token vocabulary is learned from data, favoring common patterns, explains why tokenization is efficient for typical text but can behave unexpectedly on unusual input: the model's "alphabet" of tokens is shaped by what it saw most during training, so familiar text tokenizes compactly and unfamiliar text does not.

Why Code and Other Languages Differ

Because tokenizers are optimized for the text they were trained on, typically dominated by common English prose, they tokenize other kinds of text less efficiently, which has real consequences. Code, with its unusual symbols, indentation, and identifiers, often breaks into more tokens per unit of content than plain prose, because its patterns are less common in the training data, so the same amount of code costs more tokens than the same amount of English. Text in languages underrepresented in the training data, or written in non-Latin scripts, can also tokenize into far more tokens than equivalent English, because those languages' patterns did not become efficient single tokens. This means the same idea can cost more tokens to express in one language than another, an efficiency disparity baked into the tokenizer. This is exactly why the calculator warns that code, non-English text, and unusual formatting can tokenize quite differently from typical English prose. Understanding this reveals a subtle but important consequence of tokenization: it is not neutral, but reflects the distribution of its training data, so it favors the common case and penalizes the unusual, affecting both cost and how much content fits in a model's context. The token, not the word or character, is the true unit, and its efficiency varies with how familiar the text is.

Why Tokens Are the Real Unit

The practical upshot is that tokens, not words or characters, are the fundamental unit for everything about working with language models: context limits, pricing, and processing are all measured in tokens. A model's context window, the amount of text it can consider at once, is defined in tokens, so how much text fits depends on how it tokenizes, and text that tokenizes inefficiently fills the window faster. Pricing for using models is typically per token, so the token count directly determines cost, which is why estimating tokens, as the calculator does, is the first step in budgeting. Even the model's processing works token by token. This is why the token count matters so much and why the rough word-to-token approximations the calculator uses are useful for planning, even though the exact count requires running the actual tokenizer. Understanding that tokens are the real unit reframes text from the model's perspective: what looks like a certain number of words to a human is a certain number of tokens to the model, and it is the token count that governs what fits, what it costs, and how it is processed. The calculator estimates tokens precisely because they, not the words we see, are the currency of language models.

Estimating Tokens With Understanding

Use the calculator to estimate a text's token count, and understand the tokenization behind it: models process text as subword tokens, a balance between unwieldy whole-word and overly long character approaches, the token vocabulary is learned from data so common patterns become single tokens, and code and underrepresented languages tokenize less efficiently. Tokens are the true unit of context limits, pricing, and processing. The calculation approximates the token count; understanding how tokenization works is what reveals why models don't see words, and why the token count is what really matters.