MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
  Efficient Evaluation
Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
and reasoning abilities, often assessed through multiple-choice questions
(MCQs) that include an image, a question, and several options. However, many
benchmarks used for such evaluations suffer from systematic biases. Remarkably,
Large Language Models (LLMs) without any visual perception capabilities achieve
non-trivial performance, undermining the credibility of these evaluations. To
address this issue while maintaining the efficiency of MCQ evaluations, we
propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
trilogy evaluation pipeline and more rigorous metrics. For each original
question from existing benchmarks, human annotators augment it by creating one
perception question and one knowledge anchor question through a meticulous
annotation process. MMEvalPro comprises 2,138 question triplets, totaling
6,414 distinct questions. Two-thirds of these questions are manually labeled
by human experts, while the rest are sourced from existing benchmarks (MMMU,
ScienceQA, and MathVista). Compared with the existing benchmarks, our
experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
challenging (the best LMM lags behind human performance by 31.73%, compared
to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
best LLM trails the best LMM by 23.09%, whereas the gap for previous
benchmarks is just 14.64%). Our in-depth analysis explains the reason for
the large performance gap and justifies the trustworthiness of evaluation,
underscoring its significant potential for advancing future research.
668352541d6b7ca699ee18ff
Jinsheng Huang
668352541d6b7ca699ee1900
Liang Chen
extracted_confirmed
2024-07-02T01:06:02.744Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
668352541d6b7ca699ee1901
Taian Guo
admin_assigned
2024-07-02T12:00:58.055Z
628f8ec70ea7e76254bd3c7a
/avatars/873c25eb9af052a436106a4d0618ff0b.svg
Taian Guo
user
ChazzyGordon
668352541d6b7ca699ee1902
Fu Zeng
668352541d6b7ca699ee1903
Yusheng Zhao
claimed_verified
2024-07-02T07:39:11.213Z
668388cb549c1b932c9fe699
/avatars/aa7523fbde4c2a8508cff13c74291e6a.svg
Yusheng Zhao
user
yszhao
668352541d6b7ca699ee1904
Bohan Wu
claimed_verified
2024-07-08T08:51:05.892Z
6688e31d24f96af5297860c0
/avatars/21254cbba9dd3f99df98b4a3ffc394eb.svg
Bohan Wu
user
WXT3710
668352541d6b7ca699ee1905
Ye Yuan
668352541d6b7ca699ee1906
Haozhe Zhao
claimed_verified
2024-07-22T08:43:39.664Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
668352541d6b7ca699ee1907
Zhihui Guo
668352541d6b7ca699ee1908
Yichi Zhang
668352541d6b7ca699ee1909
Jingyang Yuan
668352541d6b7ca699ee190a
Wei Ju
668352541d6b7ca699ee190b
Luchen Liu
668352541d6b7ca699ee190c
Tianyu Liu
668352541d6b7ca699ee190d
Baobao Chang
668352541d6b7ca699ee190e
Ming Zhang
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
  Efficient Evaluation
Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
and reasoning abilities, often assessed through multiple-choice questions
(MCQs) that include an image, a question, and several options. However, many
benchmarks used for such evaluations suffer from systematic biases. Remarkably,
Large Language Models (LLMs) without any visual perception capabilities achieve
non-trivial performance, undermining the credibility of these evaluations. To
address this issue while maintaining the efficiency of MCQ evaluations, we
propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
trilogy evaluation pipeline and more rigorous metrics. For each original
question from existing benchmarks, human annotators augment it by creating one
perception question and one knowledge anchor question through a meticulous
annotation process. MMEvalPro comprises 2,138 question triplets, totaling
6,414 distinct questions. Two-thirds of these questions are manually labeled
by human experts, while the rest are sourced from existing benchmarks (MMMU,
ScienceQA, and MathVista). Compared with the existing benchmarks, our
experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
challenging (the best LMM lags behind human performance by 31.73%, compared
to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
best LLM trails the best LMM by 23.09%, whereas the gap for previous
benchmarks is just 14.64%). Our in-depth analysis explains the reason for
the large performance gap and justifies the trustworthiness of evaluation,
underscoring its significant potential for advancing future research.
MMEvalPro is a benchmark designed to mitigate biases in multimodal evaluations through augmented questions and rigorous metrics, demonstrating greater challenge and trustworthiness for LMM and LLM performance assessments.
Large Multimodal Models LMMs multiple-choice questions MCQs Large Language Models LLMs perception question knowledge anchor question MMEvalPro Type-I errors triplets benchmark evaluation human performance performance gap trustworthiness