Turning on prompt caching saved me nothing—in fact, my OpenAI-API invoice jumped by about a quarter. The culprit was a single line that changed with every request: a timestamp embedded in the system prompt.

LLM providers let developers cache prompt fragments to cut token-processing costs. A cache read (a “hit”) costs as little as one-tenth of the regular rate, while a cache write (a “miss”) costs roughly 1.25 × the normal price. If a write happens but the cached fragment is never read, the extra 25 % charge is wasted. That is exactly what happened when the timestamp kept the prompt from matching any existing cache entry.

Why caching can backfire

Prompt caching works by matching the exact byte sequence of the cached portion. The provider hashes the input; if the hash matches a stored entry, the system reuses the previous computation and applies the cheap read rate. Any variation— even a single character—breaks the match and forces a fresh computation, billed at the higher write rate.

In my case the system prompt began with:

Current session started: 2026-07-14T09:41:07Z

Because the timestamp updated for every API call, the first few bytes of the request were never identical. The provider treated each call as a new cache entry, charged the write premium, and never recorded a read. The result was a steady rise in cache_creation_input_tokens while cache_read_input_tokens stayed at zero, a clear sign that the cache was never being hit.

How to spot a broken cache

The usage logs supplied by the API give two key counters:

  • cache_creation_input_tokens – tokens that triggered a write.
  • cache_read_input_tokens – tokens that benefitted from a read.

When the former climbs and the latter remains flat, the cache isn’t being reused. A quick sanity check is to repeat the exact same request twice; the second call should show a spike in read tokens if the cache is functioning.

Fixing the problem

The fix is simple: ensure the cached region is static across calls. Follow these two rules:

  1. Place immutable content first. System prompts, tool definitions, or any instruction that never changes should occupy the leading bytes of the request.
  2. Append mutable content last. Timestamps, user-generated text, request IDs, or any data that varies per call must come after the cached segment.

If even one character shifts, the hash changes and the cache miss persists. Rearranging the prompt so that the timestamp sits at the end restores the cache hit rate and brings the bill back down to the expected low-cost level.

When caching actually helps

Prompt caching shines in scenarios where the same instruction set is reused many times:

  • Agent loops where an AI repeatedly calls back to a fixed set of tools.
  • Chat sessions that reference a long, static document while only the user’s latest query changes.
  • Bulk data extraction where the same parsing prompt is applied to many records.

For single-shot calls that include a fresh context each time—like a one-off question with a unique preamble—caching offers no benefit and may even add cost if the request unintentionally triggers a write.

Hidden pitfalls

Even if the prompt itself is static, the request can be altered downstream:

  • Proxies or aggregators that re-order or inject whitespace can break the byte-for-byte match.
  • Gateway services that prepend authentication headers or modify JSON formatting may unintentionally change the cached fragment.

Testing through the gateway by sending an identical request twice and checking the read counters helps verify that the caching path remains intact.

The broader cost picture

The 25 % surcharge on writes is not a penalty for using caching; it reflects the extra compute needed to store the fragment for future reuse. When a cache hit occurs, the cost drops dramatically—often to a fraction of the regular rate. The key is to let the system actually hit the cache. Otherwise, you pay the premium without any savings.

Counter-argument: caching isn’t dead

Alcuni sviluppatori sostengono che la complessità di gestire le parti statiche rispetto a quelle dinamiche dei prompt superi il risparmio ottenuto. Questa visione trascura il fatto che molte pipeline di produzione separano già la configurazione (statica) dai dati dell'utente (dinamici). Strutturando i prompt in questo modo, è possibile sfruttare lo stesso meccanismo di caching che ha salvato gli sviluppatori originali dell'API senza sforzi aggiuntivi. Il compromesso è una modesta disciplina nel prompt design, non un difetto fondamentale della tecnologia.

Cosa monitorare successivamente

  • Monitora i due contatori di cache nella tua dashboard di utilizzo settimanalmente.
  • Effettua un audit della costruzione dei prompt per confermare che ogni elemento variabile si trovi dopo il blocco in cache.
  • Esegui test A/B con e senza caching su un carico di lavoro rappresentativo per quantificare il risparmio effettivo.
  • Valida il gateway confrontando i payload delle richieste grezze prima e dopo l'eventuale proxy.

In sintesi

Il caching dei prompt può ridurre drasticamente i costi delle API LLM, ma solo se il segmento in cache è veramente identico tra le chiamate. Un timestamp errato o qualsiasi altro token dinamico all'inizio di un prompt forza una scrittura costosa ogni volta, gonfiando la bolletta. Caricando le istruzioni statiche all'inizio e relegando i dati variabili alla fine, permetti alla cache di fare il suo lavoro e mantieni sotto controllo le spese.