MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for
  Reasoning Quality, Robustness, and Efficiency
Answering questions with Chain-of-Thought (CoT) has significantly enhanced
the reasoning capabilities of Large Language Models (LLMs), yet its impact on
Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth
investigation. In this paper, we introduce MME-CoT, a specialized benchmark
evaluating the CoT reasoning performance of LMMs, spanning six domains: math,
science, OCR, logic, space-time, and general scenes. As the first comprehensive
study in this area, we propose a thorough evaluation suite incorporating three
novel metrics that assess the reasoning quality, robustness, and efficiency at
a fine-grained level. Leveraging curated high-quality data and a unique
evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs,
uncovering several key insights: 1) Models with reflection mechanism
demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and
demonstrating the highest quality results; 2) CoT prompting often degrades LMM
performance on perception-heavy tasks, suggesting a potentially harmful
overthinking behavior; and 3) Although the CoT quality is high, LMMs with
reflection exhibit significant inefficiency in both normal response and
self-correction phases. We hope MME-CoT serves as a foundation for advancing
multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
67aee0229e69670f49533146
Dongzhi Jiang
claimed_verified
2025-02-14T08:01:05.736Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
67aee0229e69670f49533147
Renrui Zhang
67aee0229e69670f49533148
Ziyu Guo
67aee0229e69670f49533149
Yanwei Li
admin_assigned
2025-02-14T12:50:20.626Z
643ff78cdc984afcbbbc3b1a
/avatars/eec5198ce88aaf8156840bec0d190a7f.svg
Yanwei Li
user
YanweiLi
67aee0229e69670f4953314a
Yu Qi
67aee0229e69670f4953314b
Xinyan Chen
admin_assigned
2025-02-14T12:50:07.239Z
647c7a4ed412b3b376572a00
/avatars/9cc310fd3f9e3f211475816ed9b0cdaa.svg
Xinyan Chen
user
xy06
67aee0229e69670f4953314c
Liuhui Wang
67aee0229e69670f4953314d
Jianhan Jin
admin_assigned
2025-02-14T12:49:46.156Z
67aeecf31f297f2bdabd91bf
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mZBgh3oudPtx_rgSt8PJ7.png
Jianhan Jin
user
Pala718
67aee0229e69670f4953314e
Claire Guo
admin_assigned
2025-02-14T12:49:39.014Z
64862c6e19b74d6d646baa88
/avatars/5ca91ff2b7dddb36378edf74067cd9e2.svg
Claire Guo
user
clairerg
67aee0229e69670f4953314f
Shen Yan
67aee0229e69670f49533150
Bo Zhang
claimed_verified
2025-02-20T15:53:22.348Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
67aee0229e69670f49533151
Chaoyou Fu
67aee0229e69670f49533152
Peng Gao
admin_assigned
2025-02-14T12:49:22.182Z
6759af3eccbc8817f9169179
/avatars/49e64c7ccf71b8f25c52783b6ae93620.svg
Peng Gao
user
gaopenghigh
67aee0229e69670f49533153
Hongsheng Li
admin_assigned
2025-02-14T12:49:00.794Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for
  Reasoning Quality, Robustness, and Efficiency
Answering questions with Chain-of-Thought (CoT) has significantly enhanced
the reasoning capabilities of Large Language Models (LLMs), yet its impact on
Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth
investigation. In this paper, we introduce MME-CoT, a specialized benchmark
evaluating the CoT reasoning performance of LMMs, spanning six domains: math,
science, OCR, logic, space-time, and general scenes. As the first comprehensive
study in this area, we propose a thorough evaluation suite incorporating three
novel metrics that assess the reasoning quality, robustness, and efficiency at
a fine-grained level. Leveraging curated high-quality data and a unique
evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs,
uncovering several key insights: 1) Models with reflection mechanism
demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and
demonstrating the highest quality results; 2) CoT prompting often degrades LMM
performance on perception-heavy tasks, suggesting a potentially harmful
overthinking behavior; and 3) Although the CoT quality is high, LMMs with
reflection exhibit significant inefficiency in both normal response and
self-correction phases. We hope MME-CoT serves as a foundation for advancing
multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
MME-CoT provides a comprehensive evaluation of Chain-of-Thought reasoning in Large Multimodal Models across multiple domains, revealing insights into model performance, efficiency, and potential pitfalls.
Chain-of-Thought Large Language Models Large Multimodal Models MME-CoT benchmark evaluation suite reasoning quality robustness efficiency perception-heavy tasks reflection mechanism self-correction