Learn & Understand

Context Windows: Why an AI's Memory Has a Limit, and Why It's Costly

The companion calculator prices Claude API calls with separate rates for input and output tokens. Behind the input side lies a crucial concept: the context window, the amount of text a model can consider at once, which is both a hard limit and a source of real computational cost. Understanding what a context window is, why processing long context grows expensive in a particular way, why input tokens are nonetheless cheaper than output, and how caching helps turns a cost calculation into an appreciation of one of the most important constraints shaping how language models are used.

What a Context Window Is

A context window is the maximum amount of text, measured in tokens, that a model can take into account at one time when generating a response. Everything the model considers, the prompt, any instructions, documents provided, and the conversation so far, must fit within this window, and text beyond the limit cannot be considered. The context window is effectively the model's working memory for a single request: it can attend to everything within the window but has no access to anything outside it. This is why very long documents may not fit and must be trimmed or split, as the calculator's related context planning implies, and why a long conversation eventually exceeds what the model can hold. The size of the context window is a defining capability of a model, larger windows allow more text to be considered at once, enabling work with longer documents and richer context. Understanding the context window as a fixed capacity for how much text can be considered at once is fundamental: it sets the boundary of what the model can "see" for any given request, and everything you want the model to use must fit inside it.

Why Long Context Is Expensive

Processing a large context is not just a matter of fitting text in; it becomes computationally expensive in a way that grows faster than the length itself.

The mechanism that lets a model relate different parts of the text to one another, attention, works by having each part of the input consider its relationship to the other parts, and the number of such relationships grows very rapidly as the context gets longer, faster than the length itself. This means that doubling the context can more than double the computation involved in relating everything together, so very long contexts are disproportionately expensive to process. This is a core reason long context is costly and why extending context windows to great lengths has been technically challenging: the computation to attend across a huge context grows steeply. It also explains why processing a large input consumes real resources even before any output is generated. Understanding that attention makes long context expensive in a more-than-linear way clarifies why context length is both valuable and costly: the ability to consider more text at once is powerful, but the computation to do so scales unfavorably, which is why large context windows are a significant engineering and cost achievement, and why filling a context window with a huge input is not free.

Why Input Is Cheaper Than Output

Despite the cost of processing context, input tokens are priced lower than output tokens, and understanding why reconciles this with the expense of long context. When the model reads the input, it can process the entire context together in a more parallel, efficient way, taking in all the tokens at once. Generating output, by contrast, happens one token at a time, with each new token requiring a fresh pass over the context to decide what comes next, a sequential process that is more costly per token. So while a large input does consume real computation, reading it is still more efficient per token than generating output, which is done step by step. This is why the calculator applies a lower rate to input than to output: reading a token, even amid a long context, is cheaper than producing a token. Understanding this reconciles the two facts, that long context is expensive yet input is cheaper than output, the total cost of a long input is real and grows with length, but per token, reading remains less costly than the sequential work of generating. The asymmetry reflects the difference between efficiently ingesting context and laboriously producing output one piece at a time.

How Caching Helps

Because processing a large context is expensive, a valuable technique is caching, reusing the computational work done on a piece of context across multiple requests rather than repeating it. If the same large context, a long document or a fixed set of instructions, is used repeatedly across many requests, the expensive work of processing it can, in some systems, be saved and reused, so it does not have to be fully recomputed each time. This can substantially reduce the cost and time of requests that share a common, unchanging context, since the costly context processing is done once and reused rather than paid for on every call. Caching directly addresses the expense of long context by avoiding redundant work, which is especially valuable when the same context is queried many times. Understanding caching reveals a practical response to the cost of context windows: rather than paying the full price of processing a large context on every request, the work can be amortized across requests that reuse it. This is why applications that repeatedly use the same large context can be made far more economical, and it connects to the broader economics the calculator addresses, the cost of a request depends not just on token counts but on whether expensive context processing can be reused. Managing context cost, through caching and through keeping context no larger than necessary, is a key part of using models efficiently.

Pricing Requests With Context in Mind

Use the calculator to price API calls by input and output tokens, and understand the context window behind the input: it is the fixed amount of text a model can consider at once, processing long context is expensive because attention scales more than linearly with length, input tokens are still cheaper than output because reading is more efficient than generating token by token, and caching can reuse costly context processing across requests. The calculation totals input and output cost; understanding context windows is what reveals why an AI's memory has limits, why filling it is costly, and how that cost can be managed.