VMC: Video Motion Customization using Temporal Attention Adaption for
  Text-to-Video Diffusion Models
Text-to-video diffusion models have advanced video generation significantly.
However, customizing these models to generate videos with tailored motions
presents a substantial challenge. In specific, they encounter hurdles in (a)
accurately reproducing motion from a target video, and (b) creating diverse
visual variations. For example, straightforward extensions of static image
customization methods to video often lead to intricate entanglements of
appearance and motion data. To tackle this, here we present the Video Motion
Customization (VMC) framework, a novel one-shot tuning approach crafted to
adapt temporal attention layers within video diffusion models. Our approach
introduces a novel motion distillation objective using residual vectors between
consecutive frames as a motion reference. The diffusion process then preserves
low-frequency motion trajectories while mitigating high-frequency
motion-unrelated noise in image space. We validate our method against
state-of-the-art video generative models across diverse real-world motions and
contexts. Our codes, data and the project demo can be found at
https://video-motion-customization.github.io
656e90d4e9d3e5f3f7babfc6
Hyeonho Jeong
admin_assigned
2023-12-05T09:08:50.420Z
656ee8008bb9f4f8d95bd8f7
/avatars/4069d70f1279d928da521211c495d638.svg
Hyeonho Jeong
user
hyeonho-jeong-video
656e90d4e9d3e5f3f7babfc7
Geon Yeong Park
admin_assigned
2023-12-05T09:08:56.977Z
64cfff0abf39f9c8bebd2a6f
/avatars/e3405136c8af5065c287eff718c78b5b.svg
Geon Yeong Park
user
geonyoung-park
656e90d4e9d3e5f3f7babfc8
Jong Chul Ye
VMC: Video Motion Customization using Temporal Attention Adaption for
  Text-to-Video Diffusion Models
Text-to-video diffusion models have advanced video generation significantly.
However, customizing these models to generate videos with tailored motions
presents a substantial challenge. In specific, they encounter hurdles in (a)
accurately reproducing motion from a target video, and (b) creating diverse
visual variations. For example, straightforward extensions of static image
customization methods to video often lead to intricate entanglements of
appearance and motion data. To tackle this, here we present the Video Motion
Customization (VMC) framework, a novel one-shot tuning approach crafted to
adapt temporal attention layers within video diffusion models. Our approach
introduces a novel motion distillation objective using residual vectors between
consecutive frames as a motion reference. The diffusion process then preserves
low-frequency motion trajectories while mitigating high-frequency
motion-unrelated noise in image space. We validate our method against
state-of-the-art video generative models across diverse real-world motions and
contexts. Our codes, data and the project demo can be found at
https://video-motion-customization.github.io
The VMC framework enhances video generation by tuning temporal attention layers in diffusion models to accurately reproduce and diversify motion with a motion distillation objective.
text-to-video diffusion models video diffusion models temporal attention layers motion customization motion distillation residual vectors low-frequency motion trajectories high-frequency motion-unrelated noise state-of-the-art video generative models