VCR-Bench: A Comprehensive Evaluation Framework for Video
  Chain-of-Thought Reasoning
The advancement of Chain-of-Thought (CoT) reasoning has significantly
enhanced the capabilities of large language models (LLMs) and large
vision-language models (LVLMs). However, a rigorous evaluation framework for
video CoT reasoning remains absent. Current video benchmarks fail to adequately
assess the reasoning process and expose whether failures stem from deficiencies
in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a
novel benchmark designed to comprehensively evaluate LVLMs' Video
Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos
spanning a variety of video content and durations, along with 1,034
high-quality question-answer pairs. Each pair is manually annotated with a
stepwise CoT rationale, where every step is tagged to indicate its association
with the perception or reasoning capabilities. Furthermore, we design seven
distinct task dimensions and propose the CoT score to assess the entire CoT
process based on the stepwise tagged CoT rationals. Extensive experiments on
VCR-Bench highlight substantial limitations in current LVLMs. Even the
top-performing model, o1, only achieves a 62.8% CoT score and an 56.7%
accuracy, while most models score below 40%. Experiments show most models score
lower on perception than reasoning steps, revealing LVLMs' key bottleneck in
temporal-spatial information processing for complex video reasoning. A robust
positive correlation between the CoT score and accuracy confirms the validity
of our evaluation framework and underscores the critical role of CoT reasoning
in solving complex video reasoning tasks. We hope VCR-Bench to serve as a
standardized evaluation framework and expose the actual drawbacks in complex
video reasoning task.
67f886f1a0a44c8f05b7a124
Yukun Qi
admin_assigned
2025-04-11T08:24:01.756Z
66553907965ea394ee85f04c
/avatars/e0adfe83e9ec51b118e359ddc5c37a1b.svg
qyk
user
yukunqi
67f886f1a0a44c8f05b7a125
Yiming Zhao
67f886f1a0a44c8f05b7a126
Yu Zeng
claimed_verified
2025-04-14T09:47:06.461Z
665d652e0f35c005de892108
https://cdn-avatars.huggingface.co/v1/production/uploads/665d652e0f35c005de892108/OGLbgZekX-3XTBkwS8k86.jpeg
Yu Zeng
user
YuZeng260
67f886f1a0a44c8f05b7a127
Xikun Bao
admin_assigned
2025-04-11T08:24:37.813Z
672046bd4b2e5a664aca3084
/avatars/6635c617216eca67b38b64deec634fee.svg
xikun bao
user
ChthollyTree
67f886f1a0a44c8f05b7a128
Wenxuan Huang
admin_assigned
2025-04-11T08:24:49.236Z
67dc162ec8c00778e8689f42
https://cdn-avatars.huggingface.co/v1/production/uploads/67dc162ec8c00778e8689f42/_y_tO6W3ONOkOWbumAFXA.png
Wenxuan Huang
user
Osilly
67f886f1a0a44c8f05b7a129
Lin Chen
claimed_verified
2025-04-11T07:25:22.829Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
67f886f1a0a44c8f05b7a12a
Zehui Chen
admin_assigned
2025-04-11T08:24:56.718Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
user
lovesnowbest
67f886f1a0a44c8f05b7a12b
Jie Zhao
67f886f1a0a44c8f05b7a12c
Zhongang Qi
admin_assigned
2025-04-11T08:25:12.007Z
660cc64f1b14f691990c0ea0
/avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
Zhongang Qi
user
phoenixqza
67f886f1a0a44c8f05b7a12d
Feng Zhao
VCR-Bench: A Comprehensive Evaluation Framework for Video
  Chain-of-Thought Reasoning
The advancement of Chain-of-Thought (CoT) reasoning has significantly
enhanced the capabilities of large language models (LLMs) and large
vision-language models (LVLMs). However, a rigorous evaluation framework for
video CoT reasoning remains absent. Current video benchmarks fail to adequately
assess the reasoning process and expose whether failures stem from deficiencies
in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a
novel benchmark designed to comprehensively evaluate LVLMs' Video
Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos
spanning a variety of video content and durations, along with 1,034
high-quality question-answer pairs. Each pair is manually annotated with a
stepwise CoT rationale, where every step is tagged to indicate its association
with the perception or reasoning capabilities. Furthermore, we design seven
distinct task dimensions and propose the CoT score to assess the entire CoT
process based on the stepwise tagged CoT rationals. Extensive experiments on
VCR-Bench highlight substantial limitations in current LVLMs. Even the
top-performing model, o1, only achieves a 62.8% CoT score and an 56.7%
accuracy, while most models score below 40%. Experiments show most models score
lower on perception than reasoning steps, revealing LVLMs' key bottleneck in
temporal-spatial information processing for complex video reasoning. A robust
positive correlation between the CoT score and accuracy confirms the validity
of our evaluation framework and underscores the critical role of CoT reasoning
in solving complex video reasoning tasks. We hope VCR-Bench to serve as a
standardized evaluation framework and expose the actual drawbacks in complex
video reasoning task.
VCR-Bench is a new benchmark designed to evaluate video chain-of-thought reasoning in large vision-language models, assessing both perception and reasoning capabilities.
Chain-of-Thought (CoT) large language models (LLMs) large vision-language models (LVLMs) video chain-of-thought reasoning VCR-Bench stepwise CoT rationale task dimensions CoT score complex video reasoning perception reasoning temporal-spatial information processing