You paste a hundred pages of documentation into the prompt and ask a simple question. The answer comes back wrong. Or it ignores the formatting rule you gave at the top. You gave it everything. It should have worked. But it didn't.
This is the context trap. Most developers assume that feeding an AI more information automatically produces better results. The opposite is often true. Building reliable AI applications requires understanding three core mechanics: how models count text, how much they can hold at once, and what happens to information once a conversation ends.
Tokens: The Real Currency
A token is the smallest unit a language model processes. It might be a single character, a fragment of a word, or a common word in its entirety. The word “purchase” often breaks into two tokens, while a short word like “cat” stays whole. Punctuation and spaces consume tokens too. Code is especially hungry. A block of Python with nested indentation and special characters can balloon to three or four times the token count you would guess by looking at it.
Why should builders care? API pricing is per token. So is processing time. A prompt that looks like two pages of text might cost pennies or dollars depending on what is inside. Worse, tokens add up on both sides of the exchange. You pay for every token in your prompt, and you pay for every token the model generates back. Unchecked context growth quietly erodes margins.
The Context Window Is a Whiteboard
The context window sets the upper bound on what a model can see in one shot. Picture a whiteboard hanging in a small room. You can fill it with system instructions, user questions, retrieved documents, and prior conversation. But the board never grows. When new text arrives, old text falls off the edge.
This matters because models do not warn you when they start dropping content. If your system instruction sits at the top of a long conversation, and you keep appending messages, that instruction eventually scrolls out of view. The model might revert to default behavior, ignore your formatting rules, or contradict earlier guidance. Different models offer different limits, some handling a few thousand tokens and others handling hundreds of thousands, but the mechanics remain identical. Input and output share the same budget. A model that generates a two-thousand-token response has two thousand fewer tokens available for remembering what you told it.
Memory Is a Hack, Not a Feature
There is no persistent memory inside the model weights during inference. None. When you close the tab and return tomorrow, the model does not recognize you. Every API call is a cold start.
What feels like memory is just clever bookkeeping by the application layer. The frontend stores your messages in a database. When you send a new query, the software grabs the relevant history, stitches it into a fresh prompt, and ships the whole package to the model. The model itself remains oblivious.
This distinction is crucial for product teams. If you rely on the model to “remember” a user preference, you are building on sand. You need to build state management yourself. Decide what to cache. Decide how to refresh it. And recognize that every byte of history you resend eats your whiteboard space.
When Context Becomes Noise
Flooding the prompt backfires in predictable ways.
Noise kills signal. If you dump an entire codebase to ask about one bug, the model faces a needle-in-a-haystack problem. It might reference the wrong file, suggest changes to dead code, or produce a generic answer because it cannot