Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
Recent advances in CoT reasoning and RL post-training have been reported to
enhance video reasoning capabilities of MLLMs. This progress naturally raises a
question: can these models perform complex video reasoning in a manner
comparable to human experts? However, existing video benchmarks primarily
evaluate visual perception and grounding abilities, with questions that can be
answered based on explicit prompts or isolated visual cues. Such benchmarks do
not fully capture the intricacies of real-world reasoning, where humans must
actively search for, integrate, and analyze multiple clues before reaching a
conclusion. To address this issue, we present Video-Holmes, a benchmark
inspired by the reasoning process of Sherlock Holmes, designed to evaluate the
complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837
questions derived from 270 manually annotated suspense short films, which spans
seven carefully designed tasks. Each task is constructed by first identifying
key events and causal relationships within films, and then designing questions
that require models to actively locate and connect multiple relevant visual
clues scattered across different video segments. Our comprehensive evaluation
of state-of-the-art MLLMs reveals that, while these models generally excel at
visual perception, they encounter substantial difficulties with integrating
information and often miss critical clues. For example, the best-performing
model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models
scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for
multimodal reasoning, motivating models to reason more like humans and
emphasizing the ongoing challenges in this field. The benchmark is released in
https://github.com/TencentARC/Video-Holmes.
68366975d4ea32a1b4eedd82
Junhao Cheng
claimed_verified
2025-05-28T08:58:22.568Z
6506b77a773ceaa8d52ecea1
/avatars/0e769a0795063e1491c44760a4a83097.svg
CJH
user
Howe666
68366975d4ea32a1b4eedd83
Yuying Ge
admin_assigned
2025-05-28T12:23:21.224Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
68366975d4ea32a1b4eedd84
Teng Wang
claimed_verified
2025-06-05T15:08:35.735Z
63a9414e32ed73936ec0a0c8
/avatars/f181e1bb480502c2680be5296a036bdd.svg
wybertwang
user
wybertwang
68366975d4ea32a1b4eedd85
Yixiao Ge
admin_assigned
2025-05-28T12:23:47.334Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
68366975d4ea32a1b4eedd86
Jing Liao
admin_assigned
2025-05-28T12:24:02.238Z
65e77726767bfc7d109c45bf
/avatars/24e68c86e06055ea1209598ba49ce8b9.svg
Jing Liao
user
CeciliaJL
68366975d4ea32a1b4eedd87
Ying Shan
admin_assigned
2025-05-28T12:24:43.904Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
Recent advances in CoT reasoning and RL post-training have been reported to
enhance video reasoning capabilities of MLLMs. This progress naturally raises a
question: can these models perform complex video reasoning in a manner
comparable to human experts? However, existing video benchmarks primarily
evaluate visual perception and grounding abilities, with questions that can be
answered based on explicit prompts or isolated visual cues. Such benchmarks do
not fully capture the intricacies of real-world reasoning, where humans must
actively search for, integrate, and analyze multiple clues before reaching a
conclusion. To address this issue, we present Video-Holmes, a benchmark
inspired by the reasoning process of Sherlock Holmes, designed to evaluate the
complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837
questions derived from 270 manually annotated suspense short films, which spans
seven carefully designed tasks. Each task is constructed by first identifying
key events and causal relationships within films, and then designing questions
that require models to actively locate and connect multiple relevant visual
clues scattered across different video segments. Our comprehensive evaluation
of state-of-the-art MLLMs reveals that, while these models generally excel at
visual perception, they encounter substantial difficulties with integrating
information and often miss critical clues. For example, the best-performing
model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models
scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for
multimodal reasoning, motivating models to reason more like humans and
emphasizing the ongoing challenges in this field. The benchmark is released in
https://github.com/TencentARC/Video-Holmes.
Video-Holmes benchmark evaluates complex video reasoning capabilities of MLLMs using suspense short films and reveals significant challenges in information integration compared to human experts.
CoT reasoning RL post-training MLLMs Visual perception Grounding abilities Video-Holmes Suspense short films Multimodal reasoning Holmes-test