Turning on prompt caching saved me nothing—in fact, my OpenAI-API invoice jumped by about a quarter. The culprit was a single line that changed with every request: a timestamp embedded in the system prompt.

LLM providers let developers cache prompt fragments to cut token-processing costs. A cache read (a “hit”) costs as little as one-tenth of the regular rate, while a cache write (a “miss”) costs roughly 1.25 × the normal price. If a write happens but the cached fragment is never read, the extra 25 % charge is wasted. That is exactly what happened when the timestamp kept the prompt from matching any existing cache entry.

Why caching can backfire

Prompt caching works by matching the exact byte sequence of the cached portion. The provider hashes the input; if the hash matches a stored entry, the system reuses the previous computation and applies the cheap read rate. Any variation— even a single character—breaks the match and forces a fresh computation, billed at the higher write rate.

In my case the system prompt began with:

Current session started: 2026-07-14T09:41:07Z

Because the timestamp updated for every API call, the first few bytes of the request were never identical. The provider treated each call as a new cache entry, charged the write premium, and never recorded a read. The result was a steady rise in cache_creation_input_tokens while cache_read_input_tokens stayed at zero, a clear sign that the cache was never being hit.

How to spot a broken cache

The usage logs supplied by the API give two key counters:

  • cache_creation_input_tokens – tokens that triggered a write.
  • cache_read_input_tokens – tokens that benefitted from a read.

When the former climbs and the latter remains flat, the cache isn’t being reused. A quick sanity check is to repeat the exact same request twice; the second call should show a spike in read tokens if the cache is functioning.

Fixing the problem

The fix is simple: ensure the cached region is static across calls. Follow these two rules:

  1. Place immutable content first. System prompts, tool definitions, or any instruction that never changes should occupy the leading bytes of the request.
  2. Append mutable content last. Timestamps, user-generated text, request IDs, or any data that varies per call must come after the cached segment.

If even one character shifts, the hash changes and the cache miss persists. Rearranging the prompt so that the timestamp sits at the end restores the cache hit rate and brings the bill back down to the expected low-cost level.

When caching actually helps

Prompt caching shines in scenarios where the same instruction set is reused many times:

  • Agent loops where an AI repeatedly calls back to a fixed set of tools.
  • Chat sessions that reference a long, static document while only the user’s latest query changes.
  • Bulk data extraction where the same parsing prompt is applied to many records.

For single-shot calls that include a fresh context each time—like a one-off question with a unique preamble—caching offers no benefit and may even add cost if the request unintentionally triggers a write.

Hidden pitfalls

Even if the prompt itself is static, the request can be altered downstream:

  • Proxies or aggregators that re-order or inject whitespace can break the byte-for-byte match.
  • Gateway services that prepend authentication headers or modify JSON formatting may unintentionally change the cached fragment.

Testing through the gateway by sending an identical request twice and checking the read counters helps verify that the caching path remains intact.

The broader cost picture

The 25 % surcharge on writes is not a penalty for using caching; it reflects the extra compute needed to store the fragment for future reuse. When a cache hit occurs, the cost drops dramatically—often to a fraction of the regular rate. The key is to let the system actually hit the cache. Otherwise, you pay the premium without any savings.

Counter-argument: caching isn’t dead

Some developers argue that the complexity of managing static versus dynamic prompt parts outweighs the savings. That view overlooks the fact that many production pipelines already separate configuration (static) from user data (dynamic). By structuring prompts accordingly, the same caching mechanism that saved the original developers of the API can be leveraged without extra effort. The trade-off is a modest discipline in prompt design, not a fundamental flaw in the technology.

What to watch next

  • Monitor the two cache counters in your usage dashboard weekly.
  • Audit prompt construction to confirm that any variable element sits after the cached block.
  • Run A/B tests with and without caching on a representative workload to quantify actual savings.
  • Validate the gateway by comparing raw request payloads before and after any proxy.

Takeaway

Prompt caching can slash LLM API costs, but only if the cached segment is truly identical across calls. A stray timestamp or any other dynamic token at the start of a prompt forces a costly write every time, inflating the bill. By front-loading static instructions and relegating changing data to the tail end, you let the cache do its job and keep your expenses in check.