AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models
  without Specific Tuning
With the advance of text-to-image models (e.g., Stable Diffusion) and
corresponding personalization techniques such as DreamBooth and LoRA, everyone
can manifest their imagination into high-quality images at an affordable cost.
Subsequently, there is a great demand for image animation techniques to further
combine generated static images with motion dynamics. In this report, we
propose a practical framework to animate most of the existing personalized
text-to-image models once and for all, saving efforts in model-specific tuning.
At the core of the proposed framework is to insert a newly initialized motion
modeling module into the frozen text-to-image model and train it on video clips
to distill reasonable motion priors. Once trained, by simply injecting this
motion modeling module, all personalized versions derived from the same base
T2I readily become text-driven models that produce diverse and personalized
animated images. We conduct our evaluation on several public representative
personalized text-to-image models across anime pictures and realistic
photographs, and demonstrate that our proposed framework helps these models
generate temporally smooth animation clips while preserving the domain and
diversity of their outputs. Code and pre-trained weights will be publicly
available at https://animatediff.github.io/ .
64accd7cb2f83dec96103610
Yuwei Guo
claimed_verified
2023-07-12T07:29:21.009Z
6371f83d5ffa922d638ef486
https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
Yuwei Guo
user
guoyww
64accd7cb2f83dec96103611
Ceyuan Yang
claimed_verified
2023-07-11T03:45:02.111Z
64accf3fc1da7c4dbc69392d
/avatars/ec8ad2498f8a49a7a00c77a7f70d34bb.svg
Ceyuan
user
Ceyuan
64accd7cb2f83dec96103612
Anyi Rao
claimed_verified
2023-07-17T06:48:56.805Z
63f8130749569335b679af62
https://cdn-avatars.huggingface.co/v1/production/uploads/63f8130749569335b679af62/vgTu23-y0UKocwAGqNMwT.jpeg
Anyi Rao
user
anyirao
64accd7cb2f83dec96103613
Yaohui Wang
64accd7cb2f83dec96103614
Yu Qiao
64accd7cb2f83dec96103615
Dahua Lin
64accd7cb2f83dec96103616
Bo Dai
claimed_verified
2023-07-12T07:29:15.798Z
635f93577c05eb9f59966209
/avatars/add48d7e3790a6b500d6c451ef8b0f75.svg
Intelligent Digital Creation
user
BoDai
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models
  without Specific Tuning
With the advance of text-to-image models (e.g., Stable Diffusion) and
corresponding personalization techniques such as DreamBooth and LoRA, everyone
can manifest their imagination into high-quality images at an affordable cost.
Subsequently, there is a great demand for image animation techniques to further
combine generated static images with motion dynamics. In this report, we
propose a practical framework to animate most of the existing personalized
text-to-image models once and for all, saving efforts in model-specific tuning.
At the core of the proposed framework is to insert a newly initialized motion
modeling module into the frozen text-to-image model and train it on video clips
to distill reasonable motion priors. Once trained, by simply injecting this
motion modeling module, all personalized versions derived from the same base
T2I readily become text-driven models that produce diverse and personalized
animated images. We conduct our evaluation on several public representative
personalized text-to-image models across anime pictures and realistic
photographs, and demonstrate that our proposed framework helps these models
generate temporally smooth animation clips while preserving the domain and
diversity of their outputs. Code and pre-trained weights will be publicly
available at https://animatediff.github.io/ .
A framework inserts a motion model into existing text-to-image models to enable animation, using video clips to learn motion priors while preserving model domain and diversity.
text-to-image models Stable Diffusion DreamBooth LoRA motion modeling module video clips motion priors animated images