An Empirical Study of Autoregressive Pre-training from Videos
We empirically study autoregressive pre-training from videos. To perform our
study, we construct a series of autoregressive video models, called Toto. We
treat videos as sequences of visual tokens and train transformer models to
autoregressively predict future tokens. Our models are pre-trained on a diverse
dataset of videos and images comprising over 1 trillion visual tokens. We
explore different architectural, training, and inference design choices. We
evaluate the learned visual representations on a range of downstream tasks
including image recognition, video classification, object tracking, and
robotics. Our results demonstrate that, despite minimal inductive biases,
autoregressive pre-training leads to competitive performance across all
benchmarks. Finally, we find that scaling our video models results in similar
scaling curves to those seen in language models, albeit with a different rate.
More details at https://brjathu.github.io/toto/
67809ce9063dd44ffb1de7a7
Jathushan Rajasegaran
admin_assigned
2025-01-10T08:22:39.142Z
63895b3a43d8b0797a0d0406
/avatars/04d9952dff70f85d4e481ec80ae818cb.svg
Jathushan Rajasegaran
user
brjathu
67809ce9063dd44ffb1de7a8
Ilija Radosavovic
67809ce9063dd44ffb1de7a9
Rahul Ravishankar
admin_assigned
2025-01-10T08:23:24.331Z
667f3f8c5595354e745a1eb8
/avatars/7bdf766b61709e318c86795b2f48d424.svg
Rahul Ravishankar
user
rravishankar
67809ce9063dd44ffb1de7aa
Yossi Gandelsman
admin_assigned
2025-01-10T08:23:33.999Z
62f6942205ca68c0e0ff7b97
/avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
Yossi Gandelsman
user
yossig
67809ce9063dd44ffb1de7ab
Christoph Feichtenhofer
67809ce9063dd44ffb1de7ac
Jitendra Malik
admin_assigned
2025-01-10T08:24:49.391Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
An Empirical Study of Autoregressive Pre-training from Videos
We empirically study autoregressive pre-training from videos. To perform our
study, we construct a series of autoregressive video models, called Toto. We
treat videos as sequences of visual tokens and train transformer models to
autoregressively predict future tokens. Our models are pre-trained on a diverse
dataset of videos and images comprising over 1 trillion visual tokens. We
explore different architectural, training, and inference design choices. We
evaluate the learned visual representations on a range of downstream tasks
including image recognition, video classification, object tracking, and
robotics. Our results demonstrate that, despite minimal inductive biases,
autoregressive pre-training leads to competitive performance across all
benchmarks. Finally, we find that scaling our video models results in similar
scaling curves to those seen in language models, albeit with a different rate.
More details at https://brjathu.github.io/toto/
Autoregressive pre-training on a massive dataset of videos and images using transformer models yields competitive performance across various tasks and exhibits scaling behavior similar to language models.
autoregressive pre-training transformer models visual tokens image recognition video classification object tracking robotics scaling curves language models