SWE-bench scores jumped from 1.96 % to 72.7 % in under two years, a rise that headlines have framed as a 37-fold leap in AI coding ability. The headline grabs attention, but the numbers compare two different exams, not a single, steady improvement in software-engineering skill.

The raw numbers

In 2023 the original SWE-bench evaluated AI agents on 2,294 real GitHub issues. The tasks were a hodgepodge: vague descriptions, broken tests, and many problems that even a human would struggle to solve. By 2025 the same benchmark name appeared with a score of 72.7 %, but the test had been narrowed to a “Verified” subset of just 500 tasks that humans had vetted for clarity and solvability.

How the test changed

The shift from the full catalog to the Verified set is the first, most visible change. The original collection tried to reflect the chaotic reality of open-source contributions—issues that are incomplete, poorly documented, or simply impossible without additional context. The Verified version, by contrast, deliberately filters out that chaos. It presents a cleaner, more tractable problem set where high scores are realistically attainable.

Because the two versions measure different slices of the problem space, a direct percentage comparison misleads. The 1.96 % figure captures performance on raw, unfiltered work; the 72.7 % figure captures performance on a curated sample where the odds of success are far higher.

Targeted engineering

A second, subtler shift occurred in how developers approached the benchmark. In 2023, no one built agents specifically to ace SWE-bench; the test acted as a random sample of the world’s coding challenges. By 2025, teams turned the benchmark into a scoreboard. They built scaffolding, prompting strategies, and fine-tuned models with the explicit goal of scoring well on the Verified set.

When engineers design a system to pass a particular test, the score reflects how well the system fits that test, not how broadly capable it is. The benchmark stopped being a representative sample of real-world work the moment it was “repaired” and turned into a target.

What the jump really means

The headline-grabbing improvement is real in the sense that current coding agents perform dramatically better on the Verified tasks than they did on the original set. That improvement matters for competitions, research papers, and product demos that rely on the same curated benchmark.

However, the jump does not prove that AI agents can now handle the messiness of everyday software development. The original 2,294-issue set still exists, and scores on that version remain low.

Questions to ask

Whenever you see a massive swing in benchmark results, keep these three checks in mind:

  • Which version is being reported? Original, Lite, or Verified? Identical names can mask very different task pools.
  • What was filtered out? Removing noisy or impossible tasks raises the ceiling for any system; it also removes the very challenges that matter in production.
  • Was the system built to pass this specific test? If developers tuned models or pipelines for the benchmark, the score measures optimization, not raw capability.

Looking ahead

Until such safeguards become standard, the best gauge of an AI coder’s usefulness will still be its performance on the chaotic, real-world issues that developers face daily.

Takeaway: A higher score on a repaired, targeted benchmark does not automatically prove AI agents are ready for the mess of real-world code; the real test remains the unfiltered problems engineers wrestle with every day.