ExVideo: Extending Video Diffusion Models via Parameter-Efficient
  Post-Tuning
Recently, advancements in video synthesis have attracted significant
attention. Video synthesis models such as AnimateDiff and Stable Video
Diffusion have demonstrated the practical applicability of diffusion models in
creating dynamic visual content. The emergence of SORA has further spotlighted
the potential of video generation technologies. Nonetheless, the extension of
video lengths has been constrained by the limitations in computational
resources. Most existing video synthesis models can only generate short video
clips. In this paper, we propose a novel post-tuning methodology for video
synthesis models, called ExVideo. This approach is designed to enhance the
capability of current video synthesis models, allowing them to produce content
over extended temporal durations while incurring lower training expenditures.
In particular, we design extension strategies across common temporal model
architectures respectively, including 3D convolution, temporal attention, and
positional embedding. To evaluate the efficacy of our proposed post-tuning
approach, we conduct extension training on the Stable Video Diffusion model.
Our approach augments the model's capacity to generate up to 5times its
original number of frames, requiring only 1.5k GPU hours of training on a
dataset comprising 40k videos. Importantly, the substantial increase in video
length doesn't compromise the model's innate generalization capabilities, and
the model showcases its advantages in generating videos of diverse styles and
resolutions. We will release the source code and the enhanced model publicly.
6674f66fbd6e32596f69e9c8
Zhongjie Duan
admin_assigned
2024-06-21T12:08:36.125Z
6364bec8ad2c9e8b672776e6
/avatars/05293b4047437480da22f4622236d992.svg
Zhongjie Duan
user
Artiprocher
6674f66fbd6e32596f69e9c9
Wenmeng Zhou
admin_assigned
2024-06-21T12:08:42.211Z
623c6253389748c9f72ca287
https://cdn-avatars.huggingface.co/v1/production/uploads/1654828369523-623c6253389748c9f72ca287.jpeg
wenmeng zhou
user
wenmengzhou
6674f66fbd6e32596f69e9ca
Cen Chen
admin_assigned
2024-06-21T12:09:24.261Z
661e91481f06f858880080aa
/avatars/3e2a714106db5b90b6f99b0cc3b984c2.svg
chencen
user
clh13600
6674f66fbd6e32596f69e9cb
Yaliang Li
6674f66fbd6e32596f69e9cc
Weining Qian
admin_assigned
2024-06-21T12:09:06.181Z
643916835aa69077ffb27c8a
/avatars/3eeb0b8ea703d3c873839ad0fd062352.svg
Weining Qian
user
wnqian
ExVideo: Extending Video Diffusion Models via Parameter-Efficient
  Post-Tuning
Recently, advancements in video synthesis have attracted significant
attention. Video synthesis models such as AnimateDiff and Stable Video
Diffusion have demonstrated the practical applicability of diffusion models in
creating dynamic visual content. The emergence of SORA has further spotlighted
the potential of video generation technologies. Nonetheless, the extension of
video lengths has been constrained by the limitations in computational
resources. Most existing video synthesis models can only generate short video
clips. In this paper, we propose a novel post-tuning methodology for video
synthesis models, called ExVideo. This approach is designed to enhance the
capability of current video synthesis models, allowing them to produce content
over extended temporal durations while incurring lower training expenditures.
In particular, we design extension strategies across common temporal model
architectures respectively, including 3D convolution, temporal attention, and
positional embedding. To evaluate the efficacy of our proposed post-tuning
approach, we conduct extension training on the Stable Video Diffusion model.
Our approach augments the model's capacity to generate up to 5times its
original number of frames, requiring only 1.5k GPU hours of training on a
dataset comprising 40k videos. Importantly, the substantial increase in video
length doesn't compromise the model's innate generalization capabilities, and
the model showcases its advantages in generating videos of diverse styles and
resolutions. We will release the source code and the enhanced model publicly.
A novel post-tuning methodology named ExVideo enhances video synthesis models to generate longer videos with lower computational cost while maintaining quality and generalization.
diffusion models video synthesis models AnimateDiff Stable Video Diffusion SORA 3D convolution temporal attention positional embedding extension training