AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
Large-scale text-to-video (T2V) diffusion models have great progress in
recent years in terms of visual quality, motion and temporal consistency.
However, the generation process is still a black box, where all attributes
(e.g., appearance, motion) are learned and generated jointly without precise
control ability other than rough text descriptions. Inspired by image animation
which decouples the video as one specific appearance with the corresponding
motion, we propose AnimateZero to unveil the pre-trained text-to-video
diffusion model, i.e., AnimateDiff, and provide more precise appearance and
motion control abilities for it. For appearance control, we borrow intermediate
latents and their features from the text-to-image (T2I) generation for ensuring
the generated first frame is equal to the given generated image. For temporal
control, we replace the global temporal attention of the original T2V model
with our proposed positional-corrected window attention to ensure other frames
align with the first frame well. Empowered by the proposed methods, AnimateZero
can successfully control the generating progress without further training. As a
zero-shot image animator for given images, AnimateZero also enables multiple
new applications, including interactive video generation and real image
animation. The detailed experiments demonstrate the effectiveness of the
proposed method in both T2V and related applications.
6572907e990936e2e9105ec2
Jiwen Yu
admin_assigned
2023-12-08T10:09:54.581Z
64105a6d14215c0775dfdd14
https://cdn-avatars.huggingface.co/v1/production/uploads/64105a6d14215c0775dfdd14/-VX-cUYOLjHIg7QnWhRGG.jpeg
Jiwen Yu
user
VictorYuki
6572907e990936e2e9105ec3
Xiaodong Cun
admin_assigned
2023-12-08T10:10:03.681Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
6572907e990936e2e9105ec4
Chenyang Qi
admin_assigned
2023-12-08T10:10:11.359Z
642157a201c62c1e41fc526a
/avatars/73ae831a73947a04468b1e87d17da24d.svg
chenyangqi
user
chenyangqi
6572907e990936e2e9105ec5
Yong Zhang
6572907e990936e2e9105ec6
Xintao Wang
admin_assigned
2023-12-08T10:10:59.060Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
6572907e990936e2e9105ec7
Ying Shan
admin_assigned
2023-12-08T10:11:16.309Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
6572907e990936e2e9105ec8
Jian Zhang
AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
Large-scale text-to-video (T2V) diffusion models have great progress in
recent years in terms of visual quality, motion and temporal consistency.
However, the generation process is still a black box, where all attributes
(e.g., appearance, motion) are learned and generated jointly without precise
control ability other than rough text descriptions. Inspired by image animation
which decouples the video as one specific appearance with the corresponding
motion, we propose AnimateZero to unveil the pre-trained text-to-video
diffusion model, i.e., AnimateDiff, and provide more precise appearance and
motion control abilities for it. For appearance control, we borrow intermediate
latents and their features from the text-to-image (T2I) generation for ensuring
the generated first frame is equal to the given generated image. For temporal
control, we replace the global temporal attention of the original T2V model
with our proposed positional-corrected window attention to ensure other frames
align with the first frame well. Empowered by the proposed methods, AnimateZero
can successfully control the generating progress without further training. As a
zero-shot image animator for given images, AnimateZero also enables multiple
new applications, including interactive video generation and real image
animation. The detailed experiments demonstrate the effectiveness of the
proposed method in both T2V and related applications.
AnimateZero enhances pre-trained text-to-video diffusion models with precise appearance and motion control capabilities, enabling finer control over the generation process and supporting new applications like interactive video generation.
text-to-video diffusion models appearance control intermediate latents text-to-image generation temporal control global temporal attention positional-corrected window attention zero-shot image animator interactive video generation real image animation