Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short
  Videos
There has been growing sentiment recently that modern large multimodal models
(LMMs) have addressed most of the key challenges related to short video
comprehension. As a result, both academia and industry are gradually shifting
their attention towards the more complex challenges posed by understanding
long-form videos. However, is this really the case? Our studies indicate that
LMMs still lack many fundamental reasoning capabilities even when dealing with
short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation
benchmark encompassing 1000 short and natural video-caption pairs. We
demonstrate that existing LMMs severely struggle to distinguish temporal
differences between different actions and object transformations. For example,
the best model GPT-4o only obtains ~50% on our text and video scores, showing a
large gap compared to the human baseline of ~90%. All open-source multimodal
models and CLIP-based models perform much worse, producing mostly random chance
performance. Through this work, we shed light onto the fact that temporal
reasoning in short videos is a problem yet to be fully solved. The dataset and
evaluation code are available at https://vinoground.github.io.
66ff5d6c661a30ae72a1b83b
Jianrui Zhang
claimed_verified
2024-10-04T20:11:28.863Z
6508b164abdde5290e5e4939
https://cdn-avatars.huggingface.co/v1/production/uploads/6508b164abdde5290e5e4939/lQgAs3BHwCyI7Go1QA62m.jpeg
Harris Zhang
user
HanSolo9682
66ff5d6c661a30ae72a1b83c
Mu Cai
admin_assigned
2024-10-04T09:16:17.446Z
63b7b2c6bd2d153522821766
https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
Mu Cai
user
mucai
66ff5d6c661a30ae72a1b83d
Yong Jae Lee
admin_assigned
2024-10-04T09:15:57.098Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short
  Videos
There has been growing sentiment recently that modern large multimodal models
(LMMs) have addressed most of the key challenges related to short video
comprehension. As a result, both academia and industry are gradually shifting
their attention towards the more complex challenges posed by understanding
long-form videos. However, is this really the case? Our studies indicate that
LMMs still lack many fundamental reasoning capabilities even when dealing with
short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation
benchmark encompassing 1000 short and natural video-caption pairs. We
demonstrate that existing LMMs severely struggle to distinguish temporal
differences between different actions and object transformations. For example,
the best model GPT-4o only obtains ~50% on our text and video scores, showing a
large gap compared to the human baseline of ~90%. All open-source multimodal
models and CLIP-based models perform much worse, producing mostly random chance
performance. Through this work, we shed light onto the fact that temporal
reasoning in short videos is a problem yet to be fully solved. The dataset and
evaluation code are available at https://vinoground.github.io.
Vinoground is a new benchmark that reveals existing large multimodal models still lack temporal reasoning capabilities in short videos.
multimodal models LMMs temporal counterfactual video-caption pairs temporal reasoning short videos