VisOnlyQA: Large Vision Language Models Still Struggle with Visual
  Perception of Geometric Information
Errors in understanding visual information in images (i.e., visual perception
errors) remain a major source of mistakes in Large Vision Language Models
(LVLMs). While further analysis is essential, there is a deficiency in datasets
for evaluating the visual perception of LVLMs. In this work, we introduce
VisOnlyQA, a new dataset designed to directly evaluate the visual perception
capabilities of LVLMs on questions about geometric and numerical information in
scientific figures. Our dataset enables us to analyze the visual perception of
LVLMs for fine-grained visual information, independent of other capabilities
such as reasoning. The evaluation set of VisOnlyQA includes 1,200
multiple-choice questions in 12 tasks on four categories of figures. We also
provide synthetic training data consisting of 70k instances. Our experiments on
VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
synthetic training data demonstrates the potential for enhancing the visual
perception of LVLMs, but observed improvements are limited to certain tasks and
specific models. (iii) Stronger language models improve the visual perception
of LVLMs. In summary, our experiments suggest that both training data and model
architectures should be improved to enhance the visual perception capabilities
of LVLMs. The datasets, code, and model responses are provided at
https://github.com/psunlpgroup/VisOnlyQA.
674e87d63efd09e5cf7de76e
Ryo Kamoi
claimed_verified
2024-12-03T11:15:03.159Z
60591896102f61b42f65ae37
https://cdn-avatars.huggingface.co/v1/production/uploads/60591896102f61b42f65ae37/PSHcLoEeNnwvzAPOd6-6i.jpeg
Ryo Kamoi
user
ryokamoi
674e87d63efd09e5cf7de76f
Yusen Zhang
admin_assigned
2024-12-03T16:13:40.934Z
6226742d7551f131b4eba59b
/avatars/252f1fb7a82d34df4855241b5c8145ce.svg
Yusen Zhang
user
yusenzhang
674e87d63efd09e5cf7de770
Sarkar Snigdha Sarathi Das
admin_assigned
2024-12-03T16:13:46.969Z
63c707ec656e7822e23af13c
/avatars/50d2cfe1e55fe8bd2e0f09e2ca5715ea.svg
Sarkar Snigdha Sarathi Das
user
sarathismg
674e87d63efd09e5cf7de771
Ranran Haoran Zhang
674e87d63efd09e5cf7de772
Rui Zhang
VisOnlyQA: Large Vision Language Models Still Struggle with Visual
  Perception of Geometric Information
Errors in understanding visual information in images (i.e., visual perception
errors) remain a major source of mistakes in Large Vision Language Models
(LVLMs). While further analysis is essential, there is a deficiency in datasets
for evaluating the visual perception of LVLMs. In this work, we introduce
VisOnlyQA, a new dataset designed to directly evaluate the visual perception
capabilities of LVLMs on questions about geometric and numerical information in
scientific figures. Our dataset enables us to analyze the visual perception of
LVLMs for fine-grained visual information, independent of other capabilities
such as reasoning. The evaluation set of VisOnlyQA includes 1,200
multiple-choice questions in 12 tasks on four categories of figures. We also
provide synthetic training data consisting of 70k instances. Our experiments on
VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
synthetic training data demonstrates the potential for enhancing the visual
perception of LVLMs, but observed improvements are limited to certain tasks and
specific models. (iii) Stronger language models improve the visual perception
of LVLMs. In summary, our experiments suggest that both training data and model
architectures should be improved to enhance the visual perception capabilities
of LVLMs. The datasets, code, and model responses are provided at
https://github.com/psunlpgroup/VisOnlyQA.
VisOnlyQA is a novel dataset designed to evaluate the fine-grained visual perception capabilities of Large Vision Language Models (LVLMs) on scientific figures, revealing that current models perform poorly and suggesting improvements in training data and architecture are necessary.
Large Vision Language Models (LVLMs) VisOnlyQA multiple-choice questions synthetic training data visual perception fine-grained visual information GPT-4o Gemini 1.5 Pro language models