Your AI bill tripled overnight. The model, traffic volume and even the text of the prompts stayed the same; the culprit was a single line of code that broke OpenAI’s prompt cache.

Why the cache matters

The provider’s prompt cache saves you money by skipping the re-processing of any request that begins with a byte-for-byte identical prefix. If the first tokens match a prior call, the provider reuses the already-computed representation of those tokens and charges only for the new suffix. The rule is strict: the match must be exact, not merely similar. One differing token at the start destroys the whole cache hit.

The mistake that killed the hit rate

In our agent we placed a current timestamp at the very top of the system prompt to give the model a sense of “now”. Because the timestamp changes every second, the first token sequence was unique for every request. The cache never found a match, so each call incurred the full price for the 18,000 static tokens that followed—tool schemas, documentation snippets, few-shot examples and fixed instructions. The result was a 0 % cache hit rate and a bill that ballooned three-fold.

Reordering for cacheability

The fix is simple: keep everything that never changes at the front of the prompt and push any volatile data to the back.

Static prefix (cacheable)

  • Tool definitions
  • Retrieval documents
  • Few-shot examples
  • Fixed system instructions

Volatile suffix (non-cacheable)

  • Current time
  • Session identifiers
  • User messages
  • Live context

If the model needs the time, append it after the static block instead of prepending it. The cache can then reuse the heavy static portion while you still supply fresh context at the end.

Hidden killers in the stack

Even when the template looks correct, middleware or SDKs can silently prepend metadata—request IDs, timestamps or other headers—before the payload reaches the API. Some deployment pipelines also shuffle tool definitions on each rollout. Those invisible changes alter the byte sequence and sabotage the cache without any code change in your own prompt builder.

Watch the cache hit rate

Treat the cache hit rate as a primary health metric for any AI agent. A sudden dip signals that something in the request’s leading bytes has become variable. Monitoring tools that expose the hit percentage let you spot cost anomalies before they explode.

Takeaway

Prompt caching hinges on an immutable prefix. Anything that changes— even a single timestamp—at the start of every request nullifies the cache and can triple your bill. Keep static content first, volatile content last, audit your toolchain for hidden prependers, and watch cache hit rates. A disciplined prompt layout protects both performance and the bottom line.