MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
  Benchmark
This paper introduces MMMU-Pro, a robust version of the Massive
Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
MMMU-Pro rigorously assesses multimodal models' true understanding and
reasoning capabilities through a three-step process based on MMMU: (1)
filtering out questions answerable by text-only models, (2) augmenting
candidate options, and (3) introducing a vision-only input setting where
questions are embedded within images. This setting challenges AI to truly "see"
and "read" simultaneously, testing a fundamental human cognitive skill of
seamlessly integrating visual and textual information. Results show that model
performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
to 26.9% across models. We explore the impact of OCR prompts and Chain of
Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
generally improves performance. MMMU-Pro provides a more rigorous evaluation
tool, closely mimicking real-world scenarios and offering valuable directions
for future research in multimodal AI.
66d93c930d5d2187a20c381d
Xiang Yue
admin_assigned
2024-09-05T08:45:25.393Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
66d93c930d5d2187a20c381e
Tianyu Zheng
claimed_verified
2024-09-05T07:23:55.953Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
66d93c930d5d2187a20c381f
Yuansheng Ni
admin_assigned
2024-09-05T08:44:59.359Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
66d93c930d5d2187a20c3820
Yubo Wang
claimed_verified
2025-01-30T08:41:28.084Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
66d93c930d5d2187a20c3821
Kai Zhang
claimed_verified
2024-09-05T15:31:15.542Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
66d93c930d5d2187a20c3822
Shengbang Tong
66d93c930d5d2187a20c3823
Yuxuan Sun
admin_assigned
2024-09-05T08:46:44.822Z
63745f7c0938c07542400172
/avatars/df88a0b7f0693f25633240da56ce3aec.svg
yuxuan sun
user
jamessyx
66d93c930d5d2187a20c3824
Ming Yin
66d93c930d5d2187a20c3825
Botao Yu
admin_assigned
2024-09-05T08:47:50.432Z
6528d0b07b99d91baa242ca8
/avatars/56061bf1b3aca8fc49f73cbbbe38b19f.svg
Botao Yu
user
btyu
66d93c930d5d2187a20c3826
Ge Zhang
admin_assigned
2024-09-05T08:47:43.316Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66d93c930d5d2187a20c3827
Huan Sun
admin_assigned
2024-09-05T08:47:18.892Z
6127b327699a5d2cde0b44dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
Huan Sun
user
huansun
66d93c930d5d2187a20c3828
Yu Su
claimed_verified
2024-10-14T19:21:55.013Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
66d93c930d5d2187a20c3829
Wenhu Chen
admin_assigned
2024-09-05T08:47:07.238Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
66d93c930d5d2187a20c382a
Graham Neubig
admin_assigned
2024-09-05T08:47:00.965Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
  Benchmark
This paper introduces MMMU-Pro, a robust version of the Massive
Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
MMMU-Pro rigorously assesses multimodal models' true understanding and
reasoning capabilities through a three-step process based on MMMU: (1)
filtering out questions answerable by text-only models, (2) augmenting
candidate options, and (3) introducing a vision-only input setting where
questions are embedded within images. This setting challenges AI to truly "see"
and "read" simultaneously, testing a fundamental human cognitive skill of
seamlessly integrating visual and textual information. Results show that model
performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
to 26.9% across models. We explore the impact of OCR prompts and Chain of
Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
generally improves performance. MMMU-Pro provides a more rigorous evaluation
tool, closely mimicking real-world scenarios and offering valuable directions
for future research in multimodal AI.
MMMU-Pro is a benchmark that enhances the evaluation of multimodal models by filtering text-only answerable questions, augmenting options, and introducing vision-only inputs, demonstrating challenges in integrated visual and textual understanding.
MMMU-Pro multimodal models OCR prompts Chain of Thought reasoning