VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.
67cfe557ad91643b5cb7d2c6
Yanling Wang
claimed_verified
2025-03-11T16:09:28.071Z
67cd327432668b04f4555270
/avatars/15e2cef976cbe05c4c5858c88dccf4af.svg
Yanling Wang
user
WYLing
67cfe557ad91643b5cb7d2c7
Yihan Zhao
67cfe557ad91643b5cb7d2c8
Xiaodong Chen
67cfe557ad91643b5cb7d2c9
Shasha Guo
67cfe557ad91643b5cb7d2ca
Lixin Liu
67cfe557ad91643b5cb7d2cb
Haoyang Li
67cfe557ad91643b5cb7d2cc
Yong Xiao
67cfe557ad91643b5cb7d2cd
Jing Zhang
67cfe557ad91643b5cb7d2ce
Qi Li
67cfe557ad91643b5cb7d2cf
Ke Xu
VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.
Large vision-language models fact-seeking question answering multimodal benchmarks visual modality linguistic modality VisualSimpleQA VisualSimpleQA-hard GPT-4 multimodal fact-seeking QA decoupled evaluation