Learning Video Representations without Natural Videos
In this paper, we show that useful video representations can be learned from
synthetic videos and natural images, without incorporating natural videos in
the training. We propose a progression of video datasets synthesized by simple
generative processes, that model a growing set of natural video properties
(e.g. motion, acceleration, and shape transformations). The downstream
performance of video models pre-trained on these generated datasets gradually
increases with the dataset progression. A VideoMAE model pre-trained on our
synthetic videos closes 97.2% of the performance gap on UCF101 action
classification between training from scratch and self-supervised pre-training
from natural videos, and outperforms the pre-trained model on HMDB51.
Introducing crops of static images to the pre-training stage results in similar
performance to UCF101 pre-training and outperforms the UCF101 pre-trained model
on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the
low-level properties of the datasets, we identify correlations between frame
diversity, frame similarity to natural data, and downstream performance. Our
approach provides a more controllable and transparent alternative to video data
curation processes for pre-training.
6724357b3a6caef9c87e1633
Xueyang Yu
claimed_verified
2025-06-24T08:15:25.355Z
643340516c2a26ae66d5baa6
/avatars/bcf5aba7d5778160aaf5080155eafe7a.svg
Xueyang Yu
user
XueyangY
6724357b3a6caef9c87e1634
Xinlei Chen
6724357b3a6caef9c87e1635
Yossi Gandelsman
Learning Video Representations without Natural Videos
In this paper, we show that useful video representations can be learned from
synthetic videos and natural images, without incorporating natural videos in
the training. We propose a progression of video datasets synthesized by simple
generative processes, that model a growing set of natural video properties
(e.g. motion, acceleration, and shape transformations). The downstream
performance of video models pre-trained on these generated datasets gradually
increases with the dataset progression. A VideoMAE model pre-trained on our
synthetic videos closes 97.2% of the performance gap on UCF101 action
classification between training from scratch and self-supervised pre-training
from natural videos, and outperforms the pre-trained model on HMDB51.
Introducing crops of static images to the pre-training stage results in similar
performance to UCF101 pre-training and outperforms the UCF101 pre-trained model
on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the
low-level properties of the datasets, we identify correlations between frame
diversity, frame similarity to natural data, and downstream performance. Our
approach provides a more controllable and transparent alternative to video data
curation processes for pre-training.
A synthetic video dataset enables effective pre-training of VideoMAE models for video tasks without using natural videos, achieving high performance on various classification and out-of-distribution datasets.
video representations synthetic videos natural images generative processes motion acceleration shape transformations downstream performance video models VideoMAE UCF101 HMDB51 frame diversity frame similarity out-of-distribution datasets UCF101-P