VisualOverload: Probing Visual Understanding of VLMs in Really Dense
  Scenes
Is basic visual understanding really solved in state-of-the-art VLMs? We
present VisualOverload, a slightly different visual question answering (VQA)
benchmark comprising 2,720 question-answer pairs, with privately held
ground-truth responses. Unlike prior VQA datasets that typically focus on near
global image understanding, VisualOverload challenges models to perform simple,
knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our
dataset consists of high-resolution scans of public-domain paintings that are
populated with multiple figures, actions, and unfolding subplots set against
elaborately detailed backdrops. We manually annotated these images with
questions across six task categories to probe for a thorough understanding of
the scene. We hypothesize that current benchmarks overestimate the performance
of VLMs, and encoding and reasoning over details is still a challenging task
for them, especially if they are confronted with densely populated scenes.
Indeed, we observe that even the best model (o3) out of 37 tested models only
achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on
all questions. Beyond a thorough evaluation, we complement our benchmark with
an error analysis that reveals multiple failure modes, including a lack of
counting skills, failure in OCR, and striking logical inconsistencies under
complex tasks. Altogether, VisualOverload exposes a critical gap in current
vision models and offers a crucial resource for the community to develop better
models.
  Benchmark: http://paulgavrikov.github.io/visualoverload
68dc916b4159d1f2418f9aca
Paul Gavrikov
68dc916b4159d1f2418f9acb
Wei Lin
68dc916b4159d1f2418f9acc
M. Jehanzeb Mirza
68dc916b4159d1f2418f9acd
Soumya Jahagirdar
68dc916b4159d1f2418f9ace
Muhammad Huzaifa
68dc916b4159d1f2418f9acf
Sivan Doveh
68dc916b4159d1f2418f9ad0
Serena Yeung-Levy
68dc916b4159d1f2418f9ad1
James Glass
68dc916b4159d1f2418f9ad2
Hilde Kuehne
VisualOverload: Probing Visual Understanding of VLMs in Really Dense
  Scenes
Is basic visual understanding really solved in state-of-the-art VLMs? We
present VisualOverload, a slightly different visual question answering (VQA)
benchmark comprising 2,720 question-answer pairs, with privately held
ground-truth responses. Unlike prior VQA datasets that typically focus on near
global image understanding, VisualOverload challenges models to perform simple,
knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our
dataset consists of high-resolution scans of public-domain paintings that are
populated with multiple figures, actions, and unfolding subplots set against
elaborately detailed backdrops. We manually annotated these images with
questions across six task categories to probe for a thorough understanding of
the scene. We hypothesize that current benchmarks overestimate the performance
of VLMs, and encoding and reasoning over details is still a challenging task
for them, especially if they are confronted with densely populated scenes.
Indeed, we observe that even the best model (o3) out of 37 tested models only
achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on
all questions. Beyond a thorough evaluation, we complement our benchmark with
an error analysis that reveals multiple failure modes, including a lack of
counting skills, failure in OCR, and striking logical inconsistencies under
complex tasks. Altogether, VisualOverload exposes a critical gap in current
vision models and offers a crucial resource for the community to develop better
models.
  Benchmark: http://paulgavrikov.github.io/visualoverload
VisualOverload is a VQA benchmark that challenges models with simple vision tasks in densely populated scenes, revealing gaps in current VLMs' performance and offering insights into their failure modes.
visual question answering VQA densely populated scenes high-resolution scans public-domain paintings manual annotation task categories error analysis counting skills OCR logical inconsistencies vision models