IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ
  Tests
Although large Vision-Language Models (VLMs) have demonstrated remarkable
performance in a wide range of multimodal tasks, their true reasoning
capabilities on human IQ tests remain underexplored. To advance research on the
fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed
to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the
reasoning capabilities of VLMs, which we argue are more important than the
accuracy of the final prediction. **Our benchmark is visually centric,
minimizing the dependence on unnecessary textual content**, thus encouraging
models to derive answers primarily from image-based information rather than
learned textual knowledge. To this end, we manually collected and annotated 500
visual IQ questions to **prevent unintentional data leakage during training**.
Unlike prior work that focuses primarily on the accuracy of the final answer,
we evaluate the reasoning ability of the models by assessing their explanations
and the patterns used to solve each problem, along with the accuracy of the
final prediction and human evaluation. Our experiments show that there are
substantial performance disparities between tasks, with models such as
`o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest
average accuracies of 0.615, 0.578, and 0.548, respectively. However, all
models struggle with 3D spatial and anagram reasoning tasks, highlighting
significant limitations in current VLMs' general reasoning abilities. In terms
of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet`
achieved top averages of 0.696, 0.586, and 0.516, respectively. These results
highlight inconsistencies between the reasoning processes of the models and
their final answers, emphasizing the importance of evaluating the accuracy of
the reasoning in addition to the final predictions.
6838fb5e9deef11aa62251ee
Tan-Hanh Pham
6838fb5e9deef11aa62251ef
Phu-Vinh Nguyen
6838fb5e9deef11aa62251f0
Dang The Hung
6838fb5e9deef11aa62251f1
Bui Trong Duong
6838fb5e9deef11aa62251f2
Vu Nguyen Thanh
6838fb5e9deef11aa62251f3
Chris Ngo
6838fb5e9deef11aa62251f4
Tri Quang Truong
6838fb5e9deef11aa62251f5
Truong-Son Hy
IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ
  Tests
Although large Vision-Language Models (VLMs) have demonstrated remarkable
performance in a wide range of multimodal tasks, their true reasoning
capabilities on human IQ tests remain underexplored. To advance research on the
fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed
to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the
reasoning capabilities of VLMs, which we argue are more important than the
accuracy of the final prediction. **Our benchmark is visually centric,
minimizing the dependence on unnecessary textual content**, thus encouraging
models to derive answers primarily from image-based information rather than
learned textual knowledge. To this end, we manually collected and annotated 500
visual IQ questions to **prevent unintentional data leakage during training**.
Unlike prior work that focuses primarily on the accuracy of the final answer,
we evaluate the reasoning ability of the models by assessing their explanations
and the patterns used to solve each problem, along with the accuracy of the
final prediction and human evaluation. Our experiments show that there are
substantial performance disparities between tasks, with models such as
`o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest
average accuracies of 0.615, 0.578, and 0.548, respectively. However, all
models struggle with 3D spatial and anagram reasoning tasks, highlighting
significant limitations in current VLMs' general reasoning abilities. In terms
of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet`
achieved top averages of 0.696, 0.586, and 0.516, respectively. These results
highlight inconsistencies between the reasoning processes of the models and
their final answers, emphasizing the importance of evaluating the accuracy of
the reasoning in addition to the final predictions.
IQBench evaluates the reasoning capabilities of Vision-Language Models on visual IQ tests, focusing on pattern recognition and explanation quality rather than final accuracy alone.
IQBench Vision-Language Models VLMs visual IQ tests reasoning capabilities pattern recognition explanation quality