VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse
video understanding tasks with a single frozen model. We pretrain VideoPrism on
a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
approach improves upon masked autoencoding by global-local distillation of
semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
focus primarily on the video modality while leveraging the invaluable text
associated with videos. We extensively test VideoPrism on four broad groups of
video understanding tasks, from web video question answering to CV for science,
achieving state-of-the-art performance on 30 out of 33 video understanding
benchmarks.
65d573c188f4a905891d40cc
Long Zhao
extracted_confirmed
2024-02-21T04:41:13.955Z
650c249887dcda6616baa040
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
Long Zhao
user
garyzhao9012
65d573c188f4a905891d40cd
Nitesh B. Gundavarapu
65d573c188f4a905891d40ce
Liangzhe Yuan
65d573c188f4a905891d40cf
Hao Zhou
65d573c188f4a905891d40d0
Shen Yan
65d573c188f4a905891d40d1
Jennifer J. Sun
admin_assigned
2024-02-21T10:15:45.333Z
64cec85201931c6016f6f1b3
/avatars/77a4ebbb3995600cd01d385b2a6084c4.svg
Jennifer J. Sun
user
JenJSun
65d573c188f4a905891d40d2
Luke Friedman
65d573c188f4a905891d40d3
Rui Qian
65d573c188f4a905891d40d4
Tobias Weyand
65d573c188f4a905891d40d5
Yue Zhao
claimed_verified
2024-06-15T07:46:18.796Z
638fe91639f7e2a7f9d2a8c6
https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
Yue Zhao
user
zhaoyue-zephyrus
65d573c188f4a905891d40d6
Rachel Hornung
65d573c188f4a905891d40d7
Florian Schroff
65d573c188f4a905891d40d8
Ming-Hsuan Yang
65d573c188f4a905891d40d9
David A. Ross
65d573c188f4a905891d40da
Huisheng Wang
65d573c188f4a905891d40db
Hartwig Adam
65d573c188f4a905891d40dc
Mikhail Sirotenko
extracted_confirmed
2024-02-21T14:33:00.920Z
65b13272dfa3789ef39937ce
/avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg
Mikhail Sirotenko
user
msirotenko-g
65d573c188f4a905891d40dd
Ting Liu
claimed_verified
2024-02-22T22:27:59.395Z
64a78e9bacc69c3bbaf12b3d
/avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
Ting Liu
user
tingianliu
65d573c188f4a905891d40de
Boqing Gong
VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse
video understanding tasks with a single frozen model. We pretrain VideoPrism on
a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
approach improves upon masked autoencoding by global-local distillation of
semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
focus primarily on the video modality while leveraging the invaluable text
associated with videos. We extensively test VideoPrism on four broad groups of
video understanding tasks, from web video question answering to CV for science,
achieving state-of-the-art performance on 30 out of 33 video understanding
benchmarks.
VideoPrism, a pretrained video encoder, achieves top performance across various video understanding tasks by utilizing global-local distillation and token shuffling of semantic video embeddings enhanced with associated text.
VideoPrism masked autoencoding global-local distillation semantic video embeddings token shuffling scheme video understanding tasks web video question answering CV for science video understanding benchmarks