VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
We present VRBench, the first long narrative video benchmark crafted for
evaluating large models' multi-step reasoning capabilities, addressing
limitations in existing evaluations that overlook temporal reasoning and
procedural validity. It comprises 1,010 long videos (with an average duration
of 1.6 hours), along with 9,468 human-labeled multi-step question-answering
pairs and 30,292 reasoning steps with timestamps. These videos are curated via
a multi-stage filtering process including expert inter-rater reviewing to
prioritize plot coherence. We develop a human-AI collaborative framework that
generates coherent reasoning chains, each requiring multiple temporally
grounded steps, spanning seven types (e.g., event attribution, implicit
inference). VRBench designs a multi-phase evaluation pipeline that assesses
models at both the outcome and process levels. Apart from the MCQs for the
final results, we propose a progress-level LLM-guided scoring metric to
evaluate the quality of the reasoning chain from multiple dimensions
comprehensively. Through extensive evaluations of 12 LLMs and 16 VLMs on
VRBench, we undertake a thorough analysis and provide valuable insights that
advance the field of multi-step reasoning.
684b817e3b733ba333686f95
Jiashuo Yu
claimed_verified
2025-06-13T07:39:55.618Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
684b817e3b733ba333686f96
Yue Wu
684b817e3b733ba333686f97
Meng Chu
684b817e3b733ba333686f98
Zhifei Ren
684b817e3b733ba333686f99
Zizheng Huang
684b817e3b733ba333686f9a
Pei Chu
claimed_verified
2025-06-13T07:39:51.884Z
64c9beb2904317f42de06dd8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c9beb2904317f42de06dd8/he3rxfyzfwEd1vLuK6_o2.jpeg
Pei Chu
user
chupei
684b817e3b733ba333686f9b
Ruijie Zhang
684b817e3b733ba333686f9c
Yinan He
684b817e3b733ba333686f9d
Qirui Li
684b817e3b733ba333686f9e
Songze Li
claimed_verified
2025-06-13T07:39:53.776Z
64acbbd51aee69ece03c6c0c
/avatars/604df1cabc5faeda55022ae4c1997e56.svg
Songze Li
user
SongzeLi
684b817e3b733ba333686f9f
Zhenxiang Li
684b817e3b733ba333686fa0
Zhongying Tu
684b817e3b733ba333686fa1
Conghui He
684b817e3b733ba333686fa2
Yu Qiao
684b817e3b733ba333686fa3
Yali Wang
684b817e3b733ba333686fa4
Yi Wang
684b817e3b733ba333686fa5
Limin Wang
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
We present VRBench, the first long narrative video benchmark crafted for
evaluating large models' multi-step reasoning capabilities, addressing
limitations in existing evaluations that overlook temporal reasoning and
procedural validity. It comprises 1,010 long videos (with an average duration
of 1.6 hours), along with 9,468 human-labeled multi-step question-answering
pairs and 30,292 reasoning steps with timestamps. These videos are curated via
a multi-stage filtering process including expert inter-rater reviewing to
prioritize plot coherence. We develop a human-AI collaborative framework that
generates coherent reasoning chains, each requiring multiple temporally
grounded steps, spanning seven types (e.g., event attribution, implicit
inference). VRBench designs a multi-phase evaluation pipeline that assesses
models at both the outcome and process levels. Apart from the MCQs for the
final results, we propose a progress-level LLM-guided scoring metric to
evaluate the quality of the reasoning chain from multiple dimensions
comprehensively. Through extensive evaluations of 12 LLMs and 16 VLMs on
VRBench, we undertake a thorough analysis and provide valuable insights that
advance the field of multi-step reasoning.
VRBench is a long narrative video benchmark designed to evaluate models' multi-step reasoning and procedural validity through human-labeled question-answering pairs and a human-AI collaborative framework with a multi-phase evaluation pipeline.
VRBench multi-step reasoning temporal reasoning procedural validity long videos human-labeled multi-step question-answering expert inter-rater reviewing coherent reasoning chains event attribution implicit inference multi-phase evaluation progress-level LLM-guided scoring metric LLMs VLMs