Vision-language models (VLMs) still miss the mark on elementary visual tasks. A new PerceptionBench evaluation found that none of the sixteen leading systems exceeds 60 % accuracy overall, and even the strongest stays below 80 % on any individual skill. The result warns developers that high scores on popular captioning or reasoning tests do not guarantee reliable sight.

Why existing tests hide the problem

Most public benchmarks mix image description, question answering and commonsense reasoning. When a model spits out a wrong caption, the mistake could be a language slip, a reasoning error, or a simple failure to recognise the object. Because the three components are tangled, a poor score gives no clue about the visual shortfall. Teams building applications therefore take inflated confidence from headline numbers, assuming the model “sees” correctly.

What PerceptionBench does differently

PerceptionBench isolates ten visual abilities and evaluates each model on a pure-vision task set. By stripping away language and reasoning, the benchmark shows how well the visual front-end works on its own. Across the board, the sixteen top VLMs fall short of a 60 % accuracy threshold, and the best-performing system never reaches 80 % on any single skill. Hallucination—producing details that aren’t in the image—is the most common error, while the pattern of strengths and weaknesses varies widely between models.

Who stands to lose

Developers should not replace dedicated visual classifiers with general AI models.

Where the argument falters

PerceptionBench data shows diminishing returns: even the biggest models still stumble on basic perception tasks.

Practical steps for developers

  • Keep dedicated visual classifiers for tasks that demand precision; treat VLMs as complementary rather than a replacement.
  • Add a verification layer that cross-checks model outputs against a trusted detector before they reach the user.
  • Deploy a lightweight vision filter to catch obvious hallucinations or misclassifications early in the pipeline.

Pairing a general-purpose VLM with a purpose-built vision component lets teams use the former’s reasoning abilities while guarding against its visual blind spots.

Source: https://dev.to/olaughter/vision-language-models-cap-at-sixty-percent-accuracy-nj1