Can AI Resist Russian Propaganda? New Benchmark Reveals Winners
As disinformation campaigns become increasingly sophisticated, a critical question emerges: can large language models (LLMs) distinguish fact from state-sponsored manipulation? A recent study by the Institute of the Estonian Language has provided a rigorous answer, testing the resilience of 60 different AI models against targeted Russian propaganda.
The Methodology: Testing Resilience Without Web Access
To ensure the results measured the inherent reasoning capabilities of the models rather than their ability to browse the live web, researchers conducted the tests without access to search engines or external tools. The benchmark utilized 75 questions across three languages, specifically targeting 14 distinct propaganda narratives. These narratives were presented in varying degrees of difficulty, ranging from neutral phrasing to highly biased and manipulative prompts.
The evaluation process was highly structured. Each response was scored on a 1 to 5 scale, where a score of 1 indicates that the model succumbed to and repeated Russian talking points. To maintain high accuracy, a calibrated Claude Opus 4.5 served as the judge, with the results being further validated by disinformation experts from the organization Propastop.
Anthropic Leads the Pack in Disinformation Defense
The results highlight a significant performance gap between different AI providers. Anthropic’s Claude family emerged as the industry leader in resisting disinformation. Specifically, Claude Fable 5 (which is currently restricted outside the U.S.) achieved a commanding top score of 95.2. It was closely followed by Claude Opus 4.7, cementing Anthropic's position as the current gold standard for safety and factual integrity.
Other notable performers included Nvidia’s Nemotron 3 and Alibaba’s Qwen 3.6 Plus, both of which demonstrated a robust ability to identify and reject manipulative narratives.
Mistral’s Vulnerability and the Stakes for European AI
While US and Chinese models showed strength, the results were a setback for Mistral, the French AI powerhouse. Mistral's models, including the recent Medium 3.5, landed in the bottom third of the benchmark. These findings echo a previous Newsguard study that recorded a 36.67 percent misinformation rate for Mistral.
This vulnerability is particularly significant given Mistral's strategic position. As the primary European alternative to US-based and Chinese AI providers, the company is currently in the midst of negotiating a €3 billion funding round at a €20 billion valuation. For a company positioning itself as a reliable sovereign AI provider, the inability to consistently deflect propaganda presents a significant reputational and technical challenge.
Why This Matters for the AI Landscape
The stakes of this benchmark extend beyond simple accuracy scores. Russian disinformation networks, such as "Pravda," are actively working to flood AI training sets with millions of manufactured articles to "poison" the logic of future models. With OpenAI already having identified and shut down Russian campaigns using ChatGPT to influence German federal elections, the battle for the integrity of LLMs is becoming a frontline issue in global information security.
Key Takeaways
- Anthropic Dominance: Claude models, particularly Claude Fable 5, demonstrated superior resistance to propaganda compared to all other tested models.
- Mistral Under Pressure: Despite its high valuation and European importance, Mistral models struggled significantly with misinformation, trailing behind US and Chinese competitors.
- The Training Threat: The benchmark highlights the urgent need for robust defenses as state-sponsored actors actively attempt to manipulate LLM outputs through massive disinformation campaigns.