VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate existing foundation models video understanding capabilities using
a carefully designed experiment protocol consisting of three hallmark tasks
(action recognition, temporal localization, and spatiotemporal localization),
eight datasets well received by the community, and four adaptation methods
tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
adapting to general video understanding tasks. Our main findings are as
follows. First, task-specialized models significantly outperform the six FMs
studied in this work, in sharp contrast to what FMs have achieved in natural
language and image understanding. Second,video-native FMs, whose pretraining
data contains the video modality, are generally better than image-native FMs in
classifying motion-rich videos, localizing actions in time, and understanding a
video of more than one action. Third, the video-native FMs can perform well on
video tasks under light adaptations to downstream tasks(e.g., freezing the FM
backbones), while image-native FMs win in full end-to-end finetuning. The first
two observations reveal the need and tremendous opportunities to conduct
research on video-focused FMs, and the last confirms that both tasks and
adaptation methods matter when it comes to the evaluation of FMs.
64a776501e4dd9f35494e10a
Liangzhe Yuan
64a776501e4dd9f35494e10b
Nitesh Bharadwaj Gundavarapu
64a776501e4dd9f35494e10c
Long Zhao
claimed_verified
2024-10-08T07:22:57.458Z
650c249887dcda6616baa040
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
Long Zhao
user
garyzhao9012
64a776501e4dd9f35494e10d
Hao Zhou
64a776501e4dd9f35494e10e
Yin Cui
claimed_verified
2025-03-24T08:56:50.257Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
64a776501e4dd9f35494e10f
Lu Jiang
claimed_verified
2023-12-23T11:18:22.043Z
64a5cba3bea0116f8f7187a7
/avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
Lu Jiang
user
roadjiang
64a776501e4dd9f35494e110
Xuan Yang
64a776501e4dd9f35494e111
Menglin Jia
claimed_verified
2023-07-10T09:24:38.330Z
648a4681fcda61143190645e
/avatars/270bca51ef4d8453920713e9a33c46b3.svg
Menglin Jia
user
kmnp
64a776501e4dd9f35494e112
Tobias Weyand
64a776501e4dd9f35494e113
Luke Friedman
64a776501e4dd9f35494e114
Mikhail Sirotenko
claimed_verified
2024-02-21T16:30:11.812Z
65b13272dfa3789ef39937ce
/avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg
Mikhail Sirotenko
user
msirotenko-g
64a776501e4dd9f35494e115
Huisheng Wang
64a776501e4dd9f35494e116
Florian Schroff
64a776501e4dd9f35494e117
Hartwig Adam
64a776501e4dd9f35494e118
Ming-Hsuan Yang
64a776501e4dd9f35494e119
Ting Liu
claimed_verified
2023-07-10T09:24:33.909Z
64a78e9bacc69c3bbaf12b3d
/avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
Ting Liu
user
tingianliu
64a776501e4dd9f35494e11a
Boqing Gong
VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate existing foundation models video understanding capabilities using
a carefully designed experiment protocol consisting of three hallmark tasks
(action recognition, temporal localization, and spatiotemporal localization),
eight datasets well received by the community, and four adaptation methods
tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
adapting to general video understanding tasks. Our main findings are as
follows. First, task-specialized models significantly outperform the six FMs
studied in this work, in sharp contrast to what FMs have achieved in natural
language and image understanding. Second,video-native FMs, whose pretraining
data contains the video modality, are generally better than image-native FMs in
classifying motion-rich videos, localizing actions in time, and understanding a
video of more than one action. Third, the video-native FMs can perform well on
video tasks under light adaptations to downstream tasks(e.g., freezing the FM
backbones), while image-native FMs win in full end-to-end finetuning. The first
two observations reveal the need and tremendous opportunities to conduct
research on video-focused FMs, and the last confirms that both tasks and
adaptation methods matter when it comes to the evaluation of FMs.
VideoGLUE score measures foundation models' adaptation to various video tasks, revealing the superiority of video-native models over task-specialized and image-native models under different adaptions.
foundation models video understanding action recognition temporal localization spatiotemporal localization VideoGLUE score task-specialized models video-native FMs image-native FMs light adaptation end-to-end finetuning