Automated Generation of Challenging Multiple-Choice Questions for Vision
  Language Model Evaluation
The rapid development of vision language models (VLMs) demands rigorous and
reliable evaluation. However, current visual question answering (VQA)
benchmarks often depend on open-ended questions, making accurate evaluation
difficult due to the variability in natural language responses. To address
this, we introduce AutoConverter, an agentic framework that automatically
converts these open-ended questions into multiple-choice format, enabling
objective evaluation while reducing the costly question creation process. Our
experiments demonstrate that AutoConverter can generate correct and challenging
multiple-choice questions, with VLMs demonstrating consistently similar or
lower accuracy on these questions compared to human-created ones. Using
AutoConverter, we construct VMCBench, a benchmark created by transforming 20
existing VQA datasets into a unified multiple-choice format, totaling 9,018
questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
setting a new standard for scalable, consistent, and reproducible VLM
evaluation.
677ca00e34a1a97687b77c65
Yuhui Zhang
claimed_verified
2025-01-14T08:30:14.449Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
677ca00e34a1a97687b77c66
Yuchang Su
677ca00e34a1a97687b77c67
Yiming Liu
677ca00e34a1a97687b77c68
Xiaohan Wang
677ca00e34a1a97687b77c69
James Burgess
claimed_verified
2025-03-12T08:51:12.432Z
650871aeb44445e9b3625c7b
https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
James Burgess
user
jmhb
677ca00e34a1a97687b77c6a
Elaine Sui
677ca00e34a1a97687b77c6b
Chenyu Wang
677ca00e34a1a97687b77c6c
Josiah Aklilu
claimed_verified
2025-01-16T21:55:48.814Z
66844ab799dbd7c30a19a0fe
/avatars/5a5f8fa530e127105812bd1d0e515db8.svg
Josiah Aklilu
user
josaklil-ai
677ca00e34a1a97687b77c6d
Alejandro Lozano
claimed_verified
2025-01-14T08:30:12.182Z
630e56d6f6f6d700f50be105
https://cdn-avatars.huggingface.co/v1/production/uploads/630e56d6f6f6d700f50be105/_gzPXZeL2LP8oalMCriNN.jpeg
Alejandro Lozano
user
Alejandro98
677ca00e34a1a97687b77c6e
Anjiang Wei
claimed_verified
2025-04-01T07:48:08.230Z
674286496efe2b931f7ce354
/avatars/8f920618b777dbff5f2a117ebd9e9caa.svg
Anjiang Wei
user
anjiangwei
677ca00e34a1a97687b77c6f
Ludwig Schmidt
677ca00e34a1a97687b77c70
Serena Yeung-Levy
Automated Generation of Challenging Multiple-Choice Questions for Vision
  Language Model Evaluation
The rapid development of vision language models (VLMs) demands rigorous and
reliable evaluation. However, current visual question answering (VQA)
benchmarks often depend on open-ended questions, making accurate evaluation
difficult due to the variability in natural language responses. To address
this, we introduce AutoConverter, an agentic framework that automatically
converts these open-ended questions into multiple-choice format, enabling
objective evaluation while reducing the costly question creation process. Our
experiments demonstrate that AutoConverter can generate correct and challenging
multiple-choice questions, with VLMs demonstrating consistently similar or
lower accuracy on these questions compared to human-created ones. Using
AutoConverter, we construct VMCBench, a benchmark created by transforming 20
existing VQA datasets into a unified multiple-choice format, totaling 9,018
questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
setting a new standard for scalable, consistent, and reproducible VLM
evaluation.
AutoConverter converts open-ended VQA questions to multiple-choice format, enhancing evaluation of vision language models.
vision language models VLMs visual question answering VQA multiple-choice format AutoConverter VMCBench scalable evaluation reproducible evaluation