An Empirical Study of Autoregressive Pre-training from Videos
We empirically study autoregressive pre-training from videos. To perform our
study, we construct a series of autoregressive video models, called Toto. We
treat videos as sequences of visual tokens and train transformer models to
autoregressively predict future tokens. Our models are pre-trained on a diverse
dataset of videos and images comprising over 1 trillion visual tokens. We
explore different architectural, training, and inference design choices. We
evaluate the learned visual representations on a range of downstream tasks
including image recognition, video classification, object tracking, and
robotics. Our results demonstrate that, despite minimal inductive biases,
autoregressive pre-training leads to competitive performance across all
benchmarks. Finally, we find that scaling our video models results in similar
scaling curves to those seen in language models, albeit with a different rate.
More details at https://brjathu.github.io/toto/
67809ce9063dd44ffb1de7a7
Jathushan Rajasegaran
admin_assigned
2025-01-10T08:22:39.142Z
63895b3a43d8b0797a0d0406
/avatars/04d9952dff70f85d4e481ec80ae818cb.svg
Jathushan Rajasegaran
user
brjathu
67809ce9063dd44ffb1de7a8
Ilija Radosavovic
67809ce9063dd44ffb1de7a9
Rahul Ravishankar
admin_assigned
2025-01-10T08:23:24.331Z
667f3f8c5595354e745a1eb8
/avatars/7bdf766b61709e318c86795b2f48d424.svg
Rahul Ravishankar
user
rravishankar
67809ce9063dd44ffb1de7aa
Yossi Gandelsman
admin_assigned
2025-01-10T08:23:33.999Z
62f6942205ca68c0e0ff7b97
/avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
Yossi Gandelsman
user
yossig
67809ce9063dd44ffb1de7ab
Christoph Feichtenhofer
67809ce9063dd44ffb1de7ac
Jitendra Malik
admin_assigned
2025-01-10T08:24:49.391Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
An Empirical Study of Autoregressive Pre-training from Videos
We empirically study autoregressive pre-training from videos. To perform our
study, we construct a series of autoregressive video models, called Toto. We
treat videos as sequences of visual tokens and train transformer models to
autoregressively predict future tokens. Our models are pre-trained on a diverse
dataset of videos and images comprising over 1 trillion visual tokens. We
explore different architectural, training, and inference design choices. We
evaluate the learned visual representations on a range of downstream tasks
including image recognition, video classification, object tracking, and
robotics. Our results demonstrate that, despite minimal inductive biases,
autoregressive pre-training leads to competitive performance across all
benchmarks. Finally, we find that scaling our video models results in similar
scaling curves to those seen in language models, albeit with a different rate.
More details at https://brjathu.github.io/toto/
autoregressive pre-training autoregressive video models visual tokens transformer models autoregressive prediction diverse dataset architectural, training, and inference design choices visual representations image recognition video classification object tracking robotics competitive performance benchmarking scaling video models language models