Your LLM-powered agent may run perfectly in a demo, then crawl and inflate its bill after a handful of turns. The hidden culprit isn’t a flaky model—it’s token drift, the gradual bloat of the prompt that the model has to process each time.

Token drift occurs when every interaction adds more text to the model’s input context. The conversation history, tool schemas, API responses, and retrieved documents all pile up, so each subsequent call carries a larger payload. Because the model’s processing time and pricing rise with the number of input tokens, the cost climbs quadratically rather than linearly.

Why the problem shows up in production, not in demos

Real deployments preserve everything: every user utterance, every tool output, every chunk of retrieved knowledge. The accumulation stays hidden until latency spikes and the invoice arrives.

Common sources of token drift

  • Repeated transcripts – Keeping every old message in the prompt instead of summarizing or discarding it.
  • Heavy tool schemas – Sending large JSON definitions of tool capabilities on every turn.
  • Bulky tool results – Including full API responses or database rows that contain more data than the agent actually needs.
  • RAG bloat – Retrieval-augmented generation (RAG) that adds many document chunks, some of which are outdated or irrelevant.
  • Duplicated memory – Packing a summary, a state object, and the raw transcript together, which repeats the same information three times.

Each of these adds tokens that do not contribute new reasoning power, yet they inflate the prompt size.

How to keep the token budget in check

1. Adopt a layered context design

  • Stable instructions – Keep system prompts and safety rules at the top and reference them rather than resend them each turn.
  • Structured state – Store a compact representation of goals, decisions, and identifiers that the agent can read quickly.
  • Compressed history – Summarize older turns in a short, human-readable paragraph, updating only when a threshold is reached.
  • Recent turns – Include the last few messages verbatim to preserve continuity.

Separating constant text from summarizable content stops you from re-sending the same words over and over.

2. Trim tool outputs

  • Extract only the fields the agent actually uses; drop verbose descriptions.
  • Replace large results with a concise summary or a reference ID, and store the full payload in a database, cache, or blob store.
  • When a tool returns a list, send just the top-N items that matter for the current decision.

3. Apply smart summarization

  • Skip summarizing after every turn; the extra processing adds overhead.
  • Refresh the summary only when the accumulated token count of older turns crosses a preset limit.
  • Keep critical facts—IDs, amounts, timestamps—in a structured store instead of embedding them in prose, so the summary stays short.

4. Track the right metrics

  • Log token usage per model call, not just per user request. This reveals hidden growth on the input side.
  • Monitor the number of input tokens added each turn; a sudden jump points to a drift source.
  • Separate cached tokens (reused from previous calls) from newly generated tokens; only the former drive drift.

Treat the prompt as a finite resource, not an infinite transcript. By measuring, summarizing, and trimming deliberately, you keep your LLM agent fast, affordable, and ready for production scale.

Takeaway: Token drift silently drives up costs and slows agents. Identify the growing parts of your prompt, compress or externalize them, and watch token usage per call. A disciplined approach turns unpredictable bill shocks into a manageable, budget-friendly operation.