Article: Whisper versus API: When a “free” model becomes the priciest line item

Your team sees the AssemblyAI bill. Someone asks: "Why not just host Whisper ourselves to save money?"

It sounds simple. Download a checkpoint. Run a command. Done in an afternoon.

But the real question is: what is the cost to run that endpoint for the next two years?

Why the API bill looks cheap

Managed transcription services charge by the second of audio processed. AssemblyAI, for example, bills roughly $0.15 – $0.21 for every hour of content that passes through its asynchronous pipeline. Those numbers hide a lot of work: the provider maintains the inference hardware, cleans noisy audio, separates speakers, formats entities (credit-card numbers, emails, verification codes), streams results in real time, and monitors the service 24 × 7. The invoice you receive is the sum of all that, bundled into a simple per-second rate.

What the GPU price really means

Whisper Large-v3 can run on a single A100 or H100 GPU. Cloud providers list those cards at $2 – $4 per hour on demand. The catch is you pay the full hourly price even when the chip sits idle. If your workload only uses 15 % of the GPU’s capacity, you still incur the full $2 – $4 charge.

Engineering debt you inherit

Self-hosting does not stop at launching a model checkpoint. The hidden work quickly outweighs the headline hardware cost.

  • Speaker diarization – Open-source Whisper does not separate voices. Building a reliable diarization pipeline requires a separate model, data, and ongoing tuning.
  • Streaming support – Whisper processes whole files asynchronously. Real-time captions or voice-agent responses need a custom streaming layer, complete with back-pressure handling and latency monitoring.
  • Entity formatting – Raw transcripts lack the logic to mask or reformat sensitive strings. Adding rules for credit-card numbers, email addresses, or OTP codes is a non-trivial engineering effort, and a single typo can render a transcript unusable.
  • Reliability engineering – A demo that runs on one GPU is easy; a production service must juggle concurrency, retries, zero-downtime upgrades, and alerting.

When self-hosting actually pays off

The math flips only in a few narrow scenarios:

  • Heavy, continuous batch processing – If you have enough audio to keep a GPU at near-100 % utilization for months on end, the per-hour hardware charge can be amortized across a massive volume of transcription, making the per-audio cost lower than the API rate.
  • Strict data-privacy mandates – Regulations that forbid audio from leaving your premises force you to keep processing in-house, regardless of cost.
  • Full model control for prototyping – Early-stage experiments that need to tweak Whisper’s architecture, prompts, or fine-tune on proprietary data benefit from owning the checkpoint directly.

Outside those boxes, the “free” model becomes the most expensive line item because the hidden engineering debt and under-utilized GPU time quickly eclipse the modest API per-second fee.

Takeaway: Whisper’s zero-license fee is tempting, but the total cost of ownership includes GPU pricing, idle time, and a suite of engineering tasks that most teams already outsource to transcription APIs. Only when you can fully utilize the hardware, must keep data on-prem, or need deep model control does self-hosting become the cheaper route. Otherwise, the “free” model is likely the most expensive part of your transcription budget.