Title: Rename A Tool, Lose The Entire Cache

Anthropic’s new cache pricing means a single character tweak—renaming a tool or adding a timestamp to a system prompt—can turn a cheap cache hit into a full-price miss, inflating bills by dozens of times.

The shift comes from Anthropic’s two-tier cache discount. For some models a cached read costs only 0.025 × the regular input price; for others the discount is 0.1 ×. The discount applies only when the request matches a previously cached entry exactly. Misses are billed at the base rate, so the gap between hit and miss widens dramatically. In practice, a modest prompt change can push a cost up to forty times the expected amount.

Why the change matters

Anthropic added the cache tier to encourage reuse of identical prompts, a common pattern in agents that repeatedly call the same toolset. The idea is simple: store the result of a prompt-tool combination once, then retrieve it cheaply on later calls. The new multipliers make the “retrieval” side much cheaper, but they also make the penalty for a “miss” far steeper.

Developers who built agents around stable system prompts now see any drift—intentional or accidental—break the cache chain. The result is a hidden tax on the service: a lower cache-hit ratio translates directly into higher operating costs.

What breaks the cache

Anthropic’s documentation describes a hierarchy where changes at higher levels invalidate everything beneath them. The practical upshot is that seemingly harmless edits can cascade into a full-price charge.

  • Tool definitions – Adding, removing, renaming a tool, or altering its description wipes the cache for tools, the system prompt, and all message history.
  • Web-search toggle – Flipping the boolean that enables a web-search tool clears the system prompt and message cache.
  • Tool-choice parameter – Tweaking the parameter that selects which tool to run only invalidates the message cache.
  • Image payloads – Adding or removing images affects only the message cache.

Common patterns developers inadvertently trigger:

  1. Reordering tools – Some codebases sort tool dictionaries on each deployment. The new order creates a different cache key, forcing a miss every time.
  2. Timestamped prompts – Embedding a “generated at HH:MM:SS” string in the system prompt makes every request unique, guaranteeing a miss.
  3. Rotating prompt fragments – Swapping out a greeting or a version banner changes the prompt hash and busts the cache.

Spotting the hidden cost

Anthropic’s usage logs expose the cache dynamic through three fields:

  • cache_read_input_tokens – Tokens read from a cache entry.
  • cache_creation_input_tokens – Tokens that caused a new cache entry to be stored.
  • input_tokens – Tokens billed at the regular rate (the remainder after cached reads).

A sudden rise in input_tokens alongside a dip in cache_read_input_tokens signals that something in the prompt stack has shifted. Monitoring these metrics lets teams react before the bill balloons.

The developer’s response

Facing the new pricing reality, many teams now treat prompt stability as a first-class performance metric. Common strategies include:

  • Static system prompts – Store the prompt in a version-controlled file and inject it without runtime modifications.
  • Deterministic tool ordering – Define tool lists directly in code rather than relying on dictionary ordering or external generators.
  • Timestamp removal – Move logging or timing information to a separate metadata channel that does not affect the prompt string.
  • Cache-aware testing – Add unit tests that verify the hash of the full prompt (system + tools + messages) remains constant across builds.

These practices add a small engineering overhead but protect against the “hidden tax” that a cache miss now represents.

Anthropic’s perspective

Anthropic argues that the deeper discount incentivizes reuse, which can reduce overall compute load on its servers. By making cached reads dramatically cheaper, they hope developers will design agents that call the same toolset repeatedly rather than constantly reshaping prompts. The trade-off is a higher penalty for non-reusable calls, which the company says nudges developers toward better prompt hygiene.

Pengkritik menyatakan bahawa banyak ejen dunia nyata perlu menyesuaikan prompt secara langsung—menambah konteks, cap masa, atau pemilihan alatan dinamik selalunya adalah penting. Bagi beban kerja tersebut, harga baharu ini boleh menjadikan Anthropic kurang menarik berbanding penyedia yang mengenakan kadar rata tanpa mengira hit cache.

Apa yang perlu diperhatikan seterusnya

  • Semakan harga – Anthropic mungkin memperhalusi pengganda jika maklum balas komuniti menunjukkan jurang antara hit dan miss terlalu besar.
  • Ciri kawalan-cache – Kemas kini API pada masa hadapan mungkin membolehkan pembangun menentukan bahagian prompt mana yang perlu dikecualikan daripada kunci cache, sekali gus menawarkan jalan tengah.
  • Respons pesaing – Penyedia LLM lain mungkin melaraskan model cache mereka sendiri untuk kekal kompetitif, sama ada dengan menawarkan harga yang lebih rata atau dengan mendedahkan kawalan cache yang lebih terperinci.

Kesimpulan

Dengan harga cache baharu Anthropic, kos prompt miss bukan lagi sekadar kesulitan kecil; ia adalah tuas kewangan yang boleh mengubah bajet projek secara drastik. Mengekalkan prompt sistem, definisi alatan, dan metadata berkaitan agar kekal tidak berubah (immutable) kini sama pentingnya dengan menulis kod yang cekap. Pasukan yang menganggap determinisme prompt sebagai metrik yang boleh diukur akan mengelakkan bil yang mengejutkan dan kekal dalam kawalan perbelanjaan ejen AI mereka.