MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in
  Videos
The sequential structure of videos poses a challenge to the ability of
multimodal large language models (MLLMs) to locate multi-frame evidence and
conduct multimodal reasoning. However, existing video benchmarks mainly focus
on understanding tasks, which only require models to match frames mentioned in
the question (hereafter referred to as "question frame") and perceive a few
adjacent frames. To address this gap, we propose MMR-V: A Benchmark for
Multimodal Deep Reasoning in Videos. The benchmark is characterized by the
following features. (1) Long-range, multi-frame reasoning: Models are required
to infer and analyze evidence frames that may be far from the question frame.
(2) Beyond perception: Questions cannot be answered through direct perception
alone but require reasoning over hidden information. (3) Reliability: All tasks
are manually annotated, referencing extensive real-world user understanding to
align with common perceptions. (4) Confusability: Carefully designed distractor
annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos
and 1,257 tasks. Our experiments reveal that current models still struggle with
multi-modal reasoning; even the best-performing model, o4-mini, achieves only
52.5% accuracy. Additionally, current reasoning enhancement strategies
(Chain-of-Thought and scaling test-time compute) bring limited gains. Further
analysis indicates that the CoT demanded for multi-modal reasoning differs from
it in textual reasoning, which partly explains the limited performance gains.
We hope that MMR-V can inspire further research into enhancing multi-modal
reasoning capabilities.
684106fc8cb0edba3ab212bb
Kejian Zhu
claimed_verified
2025-06-11T09:23:31.242Z
650fa426877b574970bc1f0c
/avatars/55945605b53f92f715f447aab0b1ee95.svg
zkj
user
JokerJan
684106fc8cb0edba3ab212bc
Zhuoran Jin
admin_assigned
2025-06-05T12:49:35.145Z
643379416c6ecd58798421b3
/avatars/831db7eab2663abc33b176cf386b02f2.svg
Zhuoran Jin
user
jinzhuoran
684106fc8cb0edba3ab212bd
Hongbang Yuan
admin_assigned
2025-06-05T12:49:42.552Z
6538d3089d66a6c304ee2f4b
/avatars/6e93ea6c937128765e5323fb4a5d1cea.svg
HongbangYuan
user
HongbangYuan
684106fc8cb0edba3ab212be
Jiachun Li
admin_assigned
2025-06-05T12:49:50.618Z
64be6db0e38420aabaef8e12
/avatars/4d71169bdfd437a8fb9a026ac0dc46de.svg
JIACHUN LI
user
ChristinaLJC
684106fc8cb0edba3ab212bf
Shangqing Tu
admin_assigned
2025-06-05T12:49:22.312Z
648c48d8c0ddeee6df5b6d22
/avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
Shangqing Tu
user
tsq2000
684106fc8cb0edba3ab212c0
Pengfei Cao
684106fc8cb0edba3ab212c1
Yubo Chen
684106fc8cb0edba3ab212c2
Kang Liu
684106fc8cb0edba3ab212c3
Jun Zhao
MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in
  Videos
The sequential structure of videos poses a challenge to the ability of
multimodal large language models (MLLMs) to locate multi-frame evidence and
conduct multimodal reasoning. However, existing video benchmarks mainly focus
on understanding tasks, which only require models to match frames mentioned in
the question (hereafter referred to as "question frame") and perceive a few
adjacent frames. To address this gap, we propose MMR-V: A Benchmark for
Multimodal Deep Reasoning in Videos. The benchmark is characterized by the
following features. (1) Long-range, multi-frame reasoning: Models are required
to infer and analyze evidence frames that may be far from the question frame.
(2) Beyond perception: Questions cannot be answered through direct perception
alone but require reasoning over hidden information. (3) Reliability: All tasks
are manually annotated, referencing extensive real-world user understanding to
align with common perceptions. (4) Confusability: Carefully designed distractor
annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos
and 1,257 tasks. Our experiments reveal that current models still struggle with
multi-modal reasoning; even the best-performing model, o4-mini, achieves only
52.5% accuracy. Additionally, current reasoning enhancement strategies
(Chain-of-Thought and scaling test-time compute) bring limited gains. Further
analysis indicates that the CoT demanded for multi-modal reasoning differs from
it in textual reasoning, which partly explains the limited performance gains.
We hope that MMR-V can inspire further research into enhancing multi-modal
reasoning capabilities.
A new benchmark, MMR-V, is proposed to challenge multimodal large language models with long-range, multi-frame reasoning and hidden information processing in videos, revealing their limitations and inspiring further research.
multimodal large language models MMR-V long-range multi-frame reasoning multimodal reasoning manual annotation distractor annotation strategy Chain-of-Thought