Xiaomi’s new MiMo-V2-Flash model, a 309 billion-parameter mixture-of-experts (MoE) system, now tops the open-source leaderboard on SWE-Bench with 73.4 % accuracy while using less than 1/50th the compute of comparable models. The result shows elite performance is possible without the massive energy bills that have become the norm for large-language-model research.
Why Xiaomi turned to MoE
An MoE architecture sidesteps the cost by attaching many “expert” sub-networks to a shared backbone. The model stores the full 309 B parameters but activates only about 15 B for each token. This selective activation slashes inference memory and compute, letting the model run on roughly ten H100 GPUs with 618 GB of VRAM in FP16 mode.
The engineering tricks that make it practical
- Hybrid attention pattern – Most layers use sliding-window attention, limiting the attention matrix to a narrow band around each token. Every sixth layer switches to global attention, capturing long-range dependencies without blowing up memory. The pattern cuts memory use by a factor of six.
- Fast decoding module – A built-in predictor emits several tokens in a single forward pass, delivering up to 2.6× higher generation speed than standard autoregressive decoding.
- 256 K context window – The architecture ingests quarter-million-token inputs, useful for codebases, research papers, or any long document.
These components keep the model usable on hardware that would otherwise be out of reach for a 309 B-scale system.
Multi-Teacher On-Policy Distillation (MOPD)
MOPD trains the student model—MiMo-V2-Flash—using many specialized teachers: one for mathematics, another for programming, and so on. While the student generates its own responses, it receives feedback from all teachers at once. Because the student learns from the distribution it will actually produce, the distillation stays stable and the knowledge transfer remains focused on real-world tasks.
MOPD consumes less than 1/50th of the compute required by traditional methods.
Benchmark performance
| Benchmark | Score |
|---|---|
| SWE-Bench (coding) | 73.4 % |
| GPQA-Diamond (general QA) | 83.7 % |
| MMLU-Pro (multitask language understanding) | 84.9 % |
| Arena-Hard (adversarial chat) | 86.2 % |
The trade-offs that remain
Even with MoE savings, running MiMo-V2-Flash is not a laptop exercise. Deployments still need roughly ten H100 GPUs, and the 618 GB VRAM requirement confines the model to data-center environments with high-speed interconnects.
What to watch next
Takeaway: MiMo-V2-Flash proves that clever training pipelines and modular architectures can deliver top-tier performance without the runaway compute costs that have defined large-language-model development for years. The next hurdle is making that performance affordable enough for anyone beyond well-funded labs.