Why the bill exploded

When the team first added generative AI, they sent every user request to the newest, most capable model. As traffic grew, per-request costs rose in lockstep, and the CFO’s spreadsheet showed spending outpacing user growth. The usual quick fix—"just use a cheaper model"—fails in production because different queries need different reasoning levels. The real lever is how the request is sent, not which model is always used.

Building a routing layer that saves money

The engineer treated the inference service like any other production component: define tiers, set SLAs, and enforce latency budgets. The resulting architecture has four moving parts that together deliver the 95 % reduction.

Tiered routing

A thin front-end classifies each incoming request by difficulty. Roughly 95 % of queries land in a “cheap” tier that runs a modest model; only the hardest 5 % are escalated to a premium model. The classification can be rule-based (e.g., length, presence of domain-specific keywords) or learned from historical escalation data. By defaulting to the low-cost tier, the chatbot’s monthly cost fell from $420 to $28.

Model right-sizing

Matching model capability to task complexity delivers the biggest savings:

  • Simple chat – use a lightweight model instead of the flagship offering (97.5 % savings).
  • Classification – swap a mid-size model for a cheaper alternative (98.3 % savings).
  • Summarization – replace the top-tier model with a mid-range one (97.2 % savings).

The exact model names are not critical; the principle is to keep the most capable model in reserve for the few queries that truly need it.

Smart caching

Every cache hit eliminates a network call and an API charge. A distributed Redis cache stores successful responses as well as “negative” answers (“I don’t know”). When the same unanswerable question appears again, the system returns the cached “I don’t know” instead of invoking the model a second time. Over thousands of requests, this alone trims a noticeable chunk from the bill.

Prompt compression

Long prompts drive token usage, which translates directly into cost. The team runs a cheap summarizer on the client side or in a pre-processing step, shrinking a 2,000-token context to about 400 tokens before it reaches the expensive model. The token reduction multiplies across all requests, delivering massive savings without changing the end-user experience.

Strategic batching

Batching groups multiple independent requests into a single API call. The rule of thumb is simple: if a user is waiting for an answer, do not batch; if the request runs in the background (nightly reports, scheduled jobs), batch everything. Night-time batch jobs alone shave another 10-20 % off the spend.

Monitoring the optimization loop

You can’t improve what you don’t measure. The engineer set up four weekly metrics:

  1. Cost per request broken down by tier.
  2. Cache-hit rate for each routing path.
  3. Escalation rate from cheap to premium tiers.
  4. Spend per customer segment.

These numbers surface drift—e.g., a rising escalation rate may signal that the classification logic is too aggressive or that the cheap model’s quality has degraded. The team iterates on thresholds, model assignments, and cache policies each week, turning cost control into a habit rather than a crisis response.

Takeaway

A disciplined routing layer that classifies requests, right-sizes models, caches aggressively, compresses prompts, and batches background work can cut AI-API spend by up to 95 % while keeping reliability high. Treat the inference stack as a production service: define tiers, measure outcomes, and iterate weekly.