𝗛𝗼𝘄 𝗜 𝗖𝘂𝘁 𝗢𝘂𝗿 𝗔𝗜 𝗔𝗣𝗜 𝗕𝗶𝗹𝗹 𝗶𝗻 𝗛𝗮𝗹𝗳 𝗪𝗵𝗶𝗹𝗲 𝗛𝗶𝘁𝘁𝗶𝗻𝗴 𝗽𝟵𝟵 𝗦𝗟𝗔𝘀

Our AI bill was growing too fast. My CFO called it an unsustainable burn rate. At the time, we used GPT-4o for everything. It worked, but the costs were too high and the p99 latency was inconsistent.

I decided to treat AI model selection as a system design problem. I stopped looking for the best model and started looking for the best model for our specific SLAs.

I set clear targets first: • p99 latency under 1.5 seconds for chat • 99.9% availability • Multi-region failover • Throughput capacity of 3x peak load

Once I had these numbers, the solution became clear. The cheapest model per token is not always the best choice for production. If a cheap model doubles your latency, you lose users.

I compared many models. The price difference was massive. GPT-4o costs $10.00 per million output tokens. GLM-4 Plus costs $0.80. Our tests showed GLM-4 Plus performed nearly as well as GPT-4o for our specific tasks like summarization and extraction.

I built a routing layer to manage this. The system follows these rules: • Route requests based on workload type • Use a fallback model if latency spikes • Spread traffic across regions • Cache frequent requests

I also added a Redis cache. This hit rate reached 40% in one week. This reduced our token spend on repeat queries and dropped latency from 1.4 seconds to 200 milliseconds.

The results: • Monthly inference spend dropped 58% • p99 latency fell from 1.6s to 1.18s • Uptime stayed at 99.95% • Cache hit rate hit 42%

Three lessons I learned:

  1. Build your own evaluation suite. Do not trust generic benchmarks. Use your real production data.
  2. Watch rate limits closely. Regional traffic can cause unexpected spikes.
  3. Build a kill switch. A bad prompt can cause a massive spike in token usage. A cap on max tokens saved us $14,000 once.

If your AI bill is too high, define your SLA first. Build an evaluation suite from real traffic. Then, look at the pricing of models you currently ignore.

Source: https://dev.to/bolddeck/how-i-cut-our-ai-api-bill-in-half-while-hitting-p99-slas-1l05

Optional learning community: https://t.me/GyaanSetuAi