V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
Human processes video reasoning in a sequential spatio-temporal reasoning
logic, we first identify the relevant frames ("when") and then analyse the
spatial relationships ("where") between key objects, and finally leverage these
relationships to draw inferences ("what"). However, can Video Large Language
Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in
videos? Existing Video-LLM benchmarks primarily focus on assessing object
presence, neglecting relational reasoning. Consequently, it is difficult to
measure whether a model truly comprehends object interactions (actions/events)
in videos or merely relies on pre-trained "memory" of co-occurrences as biases
in generating answers. In this work, we introduce a Video Spatio-Temporal
Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to
decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR)
task that simultaneously evaluates what objects are present, when events occur,
and where they are located while capturing the underlying Chain-of-thought
(CoT) logic. To support this evaluation, we construct a dataset to elicit the
spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine
CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding
explicit reasoning chains to mimic human cognition. Experiments from 14
Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and
the needs for robust and consistent spatio-temporal reasoning.
67d8ca56e94f1237cb3ba3ca
Zixu Cheng
admin_assigned
2025-03-18T09:31:58.589Z
667ee096b0fad0fdee319ed4
/avatars/d9df687e8522d47f7fcefe40fd9b575b.svg
Zixu Cheng
user
Cade921
67d8ca56e94f1237cb3ba3cb
Jian Hu
claimed_verified
2025-03-18T08:08:20.643Z
65e1b6e9501590df0173cbd3
/avatars/a73e2139700e23eff455734c99cef5ba.svg
Jian Hu
user
lwpyh
67d8ca56e94f1237cb3ba3cc
Ziquan Liu
67d8ca56e94f1237cb3ba3cd
Chenyang Si
admin_assigned
2025-03-18T09:32:21.924Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
67d8ca56e94f1237cb3ba3ce
Wei Li
67d8ca56e94f1237cb3ba3cf
Shaogang Gong
V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
Human processes video reasoning in a sequential spatio-temporal reasoning
logic, we first identify the relevant frames ("when") and then analyse the
spatial relationships ("where") between key objects, and finally leverage these
relationships to draw inferences ("what"). However, can Video Large Language
Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in
videos? Existing Video-LLM benchmarks primarily focus on assessing object
presence, neglecting relational reasoning. Consequently, it is difficult to
measure whether a model truly comprehends object interactions (actions/events)
in videos or merely relies on pre-trained "memory" of co-occurrences as biases
in generating answers. In this work, we introduce a Video Spatio-Temporal
Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to
decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR)
task that simultaneously evaluates what objects are present, when events occur,
and where they are located while capturing the underlying Chain-of-thought
(CoT) logic. To support this evaluation, we construct a dataset to elicit the
spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine
CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding
explicit reasoning chains to mimic human cognition. Experiments from 14
Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and
the needs for robust and consistent spatio-temporal reasoning.
A new Video Spatio-Temporal Reasoning (V-STaR) benchmark assesses Video Large Language Models' ability to understand and reason through spatio-temporal relationships in videos.
Video Large Language Models (Video-LLMs) Video Spatio-Temporal Reasoning (V-STaR) Reverse Spatio-Temporal Reasoning (RSTR) Chain-of-thought (CoT)