One-Minute Video Generation with Test-Time Training
Transformers today still struggle to generate one-minute videos because
self-attention layers are inefficient for long context. Alternatives such as
Mamba layers struggle with complex multi-scene stories because their hidden
states are less expressive. We experiment with Test-Time Training (TTT) layers,
whose hidden states themselves can be neural networks, therefore more
expressive. Adding TTT layers into a pre-trained Transformer enables it to
generate one-minute videos from text storyboards. For proof of concept, we
curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
generate much more coherent videos that tell complex stories, leading by 34 Elo
points in a human evaluation of 100 videos per method. Although promising,
results still contain artifacts, likely due to the limited capability of the
pre-trained 5B model. The efficiency of our implementation can also be
improved. We have only experimented with one-minute videos due to resource
constraints, but the approach can be extended to longer videos and more complex
stories. Sample videos, code and annotations are available at:
https://test-time-training.github.io/video-dit
67f4a0ccfefcc542f83f84e7
Karan Dalal
admin_assigned
2025-04-08T09:09:24.283Z
646ebf2298e8f749fc60ce38
/avatars/5a76c9343451c24e9697d3165cdc0af6.svg
Karan Dalal
user
karansdalal
67f4a0ccfefcc542f83f84e8
Daniel Koceja
admin_assigned
2025-04-08T09:09:31.602Z
66b01b126038fe024ae979d3
/avatars/75b561be803fa28cb78c6cff9eb5ac54.svg
Daniel Koceja
user
koceja
67f4a0ccfefcc542f83f84e9
Gashon Hussein
admin_assigned
2025-04-08T09:09:39.483Z
638835fede1fa6adc5485a05
https://cdn-avatars.huggingface.co/v1/production/uploads/1669871090087-noauth.jpeg
Gashon Hussein
user
GashonHussein
67f4a0ccfefcc542f83f84ea
Jiarui Xu
claimed_verified
2025-04-08T20:06:33.532Z
61f7671a481b345bd1059092
https://cdn-avatars.huggingface.co/v1/production/uploads/1643603726045-noauth.jpeg
Jiarui Xu
user
xvjiarui
67f4a0ccfefcc542f83f84eb
Yue Zhao
claimed_verified
2025-04-08T06:51:29.995Z
638fe91639f7e2a7f9d2a8c6
https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
Yue Zhao
user
zhaoyue-zephyrus
67f4a0ccfefcc542f83f84ec
Youjin Song
67f4a0ccfefcc542f83f84ed
Shihao Han
67f4a0ccfefcc542f83f84ee
Ka Chun Cheung
67f4a0ccfefcc542f83f84ef
Jan Kautz
67f4a0ccfefcc542f83f84f0
Carlos Guestrin
admin_assigned
2025-04-08T09:10:26.609Z
677c45511fbb93b90f1c6f3d
/avatars/3a78fdd8d1debc8d267e80e4b7a6bf77.svg
Carlos Guestrin
user
guestrin
67f4a0ccfefcc542f83f84f1
Tatsunori Hashimoto
admin_assigned
2025-04-08T09:10:20.291Z
67ecbb7eb57a7b083182ea3f
/avatars/8fb800e0729771e61ff0d2f9a05eb5b9.svg
Tatsunori Hashimoto
user
hashtag56
67f4a0ccfefcc542f83f84f2
Sanmi Koyejo
admin_assigned
2025-04-08T09:10:14.122Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
67f4a0ccfefcc542f83f84f3
Yejin Choi
admin_assigned
2025-04-08T09:10:07.480Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
67f4a0ccfefcc542f83f84f4
Yu Sun
67f4a0ccfefcc542f83f84f5
Xiaolong Wang
One-Minute Video Generation with Test-Time Training
Transformers today still struggle to generate one-minute videos because
self-attention layers are inefficient for long context. Alternatives such as
Mamba layers struggle with complex multi-scene stories because their hidden
states are less expressive. We experiment with Test-Time Training (TTT) layers,
whose hidden states themselves can be neural networks, therefore more
expressive. Adding TTT layers into a pre-trained Transformer enables it to
generate one-minute videos from text storyboards. For proof of concept, we
curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
generate much more coherent videos that tell complex stories, leading by 34 Elo
points in a human evaluation of 100 videos per method. Although promising,
results still contain artifacts, likely due to the limited capability of the
pre-trained 5B model. The efficiency of our implementation can also be
improved. We have only experimented with one-minute videos due to resource
constraints, but the approach can be extended to longer videos and more complex
stories. Sample videos, code and annotations are available at:
https://test-time-training.github.io/video-dit
Test-Time Training (TTT) layers enable pre-trained Transformers to generate coherent one-minute videos from text storyboards, outperforming alternatives like Mamba~2 and Gated DeltaNet.
self-attention layers Mamba layers Test-Time Training (TTT) layers pre-trained Transformer one-minute videos text storyboards Tom and Jerry cartoons Elo points sliding-window attention layers