Towards Video Thinking Test: A Holistic Benchmark for Advanced Video
  Reasoning and Understanding
Human intelligence requires correctness and robustness, with the former being
foundational for the latter. In video understanding, correctness ensures the
accurate interpretation of visual content, and robustness maintains consistent
performance in challenging conditions. Despite advances in video large language
models (video LLMs), existing benchmarks inadequately reflect the gap between
these models and human intelligence in maintaining correctness and robustness
in video interpretation. We introduce the Video Thinking Test (Video-TT), to
assess if video LLMs can interpret real-world videos as effectively as humans.
Video-TT reflects genuine gaps in understanding complex visual narratives, and
evaluates robustness against natural adversarial questions. Video-TT comprises
1,000 YouTube Shorts videos, each with one open-ended question and four
adversarial questions that probe visual and narrative complexity. Our
evaluation shows a significant gap between video LLMs and human performance.
687f104d33947f780d9b4b35
Yuanhan Zhang
claimed_verified
2025-07-22T07:47:48.155Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
687f104d33947f780d9b4b36
Yunice Chew
claimed_verified
2025-07-22T07:47:45.878Z
66e6cf54c86016ecd9ee9ea7
https://cdn-avatars.huggingface.co/v1/production/uploads/66e6cf54c86016ecd9ee9ea7/2UAIrV4aCHeBrMg-R1-gF.jpeg
yunice
user
yunicechew
687f104d33947f780d9b4b37
Yuhao Dong
claimed_verified
2025-07-25T08:39:19.750Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
687f104d33947f780d9b4b38
Aria Leo
687f104d33947f780d9b4b39
Bo Hu
687f104d33947f780d9b4b3a
Ziwei Liu
Towards Video Thinking Test: A Holistic Benchmark for Advanced Video
  Reasoning and Understanding
Human intelligence requires correctness and robustness, with the former being
foundational for the latter. In video understanding, correctness ensures the
accurate interpretation of visual content, and robustness maintains consistent
performance in challenging conditions. Despite advances in video large language
models (video LLMs), existing benchmarks inadequately reflect the gap between
these models and human intelligence in maintaining correctness and robustness
in video interpretation. We introduce the Video Thinking Test (Video-TT), to
assess if video LLMs can interpret real-world videos as effectively as humans.
Video-TT reflects genuine gaps in understanding complex visual narratives, and
evaluates robustness against natural adversarial questions. Video-TT comprises
1,000 YouTube Shorts videos, each with one open-ended question and four
adversarial questions that probe visual and narrative complexity. Our
evaluation shows a significant gap between video LLMs and human performance.
Video-TT assesses video LLMs' correctness and robustness in interpreting real-world videos through open-ended and adversarial questions.
video large language models video LLMs Video Thinking Test Video-TT visual narratives adversarial questions