Pandora's AI Model Routing Box
DeepMind’s new paper proposes a “Pandora’s Box” framework that cuts routing overhead by up to 70 % while keeping task performance unchanged.
Why routing costs matter
When a service gets a request, it often has a menu of models—some fast but coarse, others slow but precise. Picking the best one costs compute and dollars. You must run a quick test, fetch external data, or call a helper tool before you know which model fits.
The Pandora’s Box analogy
DeepMind maps the routing problem onto the classic Pandora’s Box puzzle: each model is a sealed box hiding its performance on the current query. Opening a box consumes resources, so you must decide if the potential gain justifies the expense. The framework formalizes this trade-off and provides two concrete mechanisms.
Pandora’s Router – a centralized decision maker
The router first runs a cheap, noisy estimator to gauge a model’s promise. Only if the expected improvement over the current best exceeds a preset “reservation price” does it pay for a more accurate assessment. By refusing to inspect every model, the router slashes the inspection budget while still surfacing the top-performing option.
Pandora’s Bidder – a decentralized marketplace
In the bidder variant, each model provider decides whether to spend money on a self-assessment that could win a contract. Providers submit bids only when they expect the assessment cost to be outweighed by the chance of winning. This creates a market where costly evaluations happen only when the payoff is high enough.
Results across three domains
The authors tested both mechanisms on:
- Math reasoning – selecting the model that solves a problem most accurately.
- Retrieval-augmented generation (RAG) – choosing the retrieval system that supplies the most relevant context before a language model generates an answer.
- Large-scale embedding selection – picking the encoder that yields the best vector representation for similarity search.
In every case, Pandora’s Router recorded the lowest combined cost-plus-error metric, beating baselines that always inspect every model or never inspect at all. The system adapts automatically: when inspection costs rise, it inspects fewer models; when they fall, it ramps up scrutiny. Reported savings range from 30 % to 70 % of routing expenses, with no measurable drop in downstream task quality.
Takeaway: As inference budgets balloon, deciding when to stop probing for a better model becomes as crucial as choosing the model itself. Pandora’s Box turns a hidden expense into a controllable lever.