VideoTetris: Towards Compositional Text-to-Video Generation
Diffusion models have demonstrated great success in text-to-video (T2V)
generation. However, existing methods may face challenges when handling complex
(long) video generation scenarios that involve multiple objects or dynamic
changes in object numbers. To address these limitations, we propose
VideoTetris, a novel framework that enables compositional T2V generation.
Specifically, we propose spatio-temporal compositional diffusion to precisely
follow complex textual semantics by manipulating and composing the attention
maps of denoising networks spatially and temporally. Moreover, we propose an
enhanced video data preprocessing to enhance the training data regarding motion
dynamics and prompt understanding, equipped with a new reference frame
attention mechanism to improve the consistency of auto-regressive video
generation. Extensive experiments demonstrate that our VideoTetris achieves
impressive qualitative and quantitative results in compositional T2V
generation. Code is available at: https://github.com/YangLing0818/VideoTetris
6662687368009120aff96b94
Ye Tian
claimed_verified
2025-03-25T08:23:30.477Z
64e357dd825f4133e7427bf8
https://cdn-avatars.huggingface.co/v1/production/uploads/64e357dd825f4133e7427bf8/HwaWhINrzkbXG6SHG2oyf.jpeg
tyfeld
user
tyfeld
6662687368009120aff96b95
Ling Yang
claimed_verified
2024-07-12T07:41:56.636Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
6662687368009120aff96b96
Haotian Yang
admin_assigned
2024-06-07T10:17:30.822Z
64ba53dcbc787364968a7649
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KgbDlawW8n2s_V3kzhJLW.png
Haotian Yang
user
haotianxoxo
6662687368009120aff96b97
Yuan Gao
6662687368009120aff96b98
Yufan Deng
6662687368009120aff96b99
Jingmin Chen
admin_assigned
2024-06-07T10:03:28.754Z
64218b51e61513ec1c63d6d4
/avatars/2a14f872a391737b3d862788ab9f99df.svg
Jingmin Chen 
user
Jingmin
6662687368009120aff96b9a
Xintao Wang
admin_assigned
2024-06-07T10:03:20.223Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
6662687368009120aff96b9b
Zhaochen Yu
6662687368009120aff96b9c
Xin Tao
6662687368009120aff96b9d
Pengfei Wan
6662687368009120aff96b9e
Di Zhang
admin_assigned
2024-06-07T09:46:13.604Z
6306ec4f1e731505a77646f0
/avatars/5d065dd89324072176cc29498cd98b03.svg
ZhangDi
user
ZhangDi
6662687368009120aff96b9f
Bin Cui
VideoTetris: Towards Compositional Text-to-Video Generation
Diffusion models have demonstrated great success in text-to-video (T2V)
generation. However, existing methods may face challenges when handling complex
(long) video generation scenarios that involve multiple objects or dynamic
changes in object numbers. To address these limitations, we propose
VideoTetris, a novel framework that enables compositional T2V generation.
Specifically, we propose spatio-temporal compositional diffusion to precisely
follow complex textual semantics by manipulating and composing the attention
maps of denoising networks spatially and temporally. Moreover, we propose an
enhanced video data preprocessing to enhance the training data regarding motion
dynamics and prompt understanding, equipped with a new reference frame
attention mechanism to improve the consistency of auto-regressive video
generation. Extensive experiments demonstrate that our VideoTetris achieves
impressive qualitative and quantitative results in compositional T2V
generation. Code is available at: https://github.com/YangLing0818/VideoTetris
VideoTetris framework uses compositional diffusion and enhanced preprocessing to generate complex text-to-video content with improved consistency and dynamic handling.
diffusion models text-to-video spatio-temporal compositional diffusion denoising networks attention maps video data preprocessing reference frame attention mechanism auto-regressive video generation