CustomCrafter: Customized Video Generation with Preserving Motion and
  Concept Composition Abilities
Customized video generation aims to generate high-quality videos guided by
text prompts and subject's reference images. However, since it is only trained
on static images, the fine-tuning process of subject learning disrupts
abilities of video diffusion models (VDMs) to combine concepts and generate
motions. To restore these abilities, some methods use additional video similar
to the prompt to fine-tune or guide the model. This requires frequent changes
of guiding videos and even re-tuning of the model when generating different
motions, which is very inconvenient for users. In this paper, we propose
CustomCrafter, a novel framework that preserves the model's motion generation
and conceptual combination abilities without additional video and fine-tuning
to recovery. For preserving conceptual combination ability, we design a
plug-and-play module to update few parameters in VDMs, enhancing the model's
ability to capture the appearance details and the ability of concept
combinations for new subjects. For motion generation, we observed that VDMs
tend to restore the motion of video in the early stage of denoising, while
focusing on the recovery of subject details in the later stage. Therefore, we
propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our
subject learning modules, we reduce the impact of this module on motion
generation in the early stage of denoising, preserving the ability to generate
motion of VDMs. In the later stage of denoising, we restore this module to
repair the appearance details of the specified subject, thereby ensuring the
fidelity of the subject's appearance. Experimental results show that our method
has a significant improvement compared to previous methods.
66cbe22f8347e21e8a75a852
Tao Wu
66cbe22f8347e21e8a75a853
Yong Zhang
66cbe22f8347e21e8a75a854
Xintao Wang
66cbe22f8347e21e8a75a855
Xianpan Zhou
66cbe22f8347e21e8a75a856
Guangcong Zheng
66cbe22f8347e21e8a75a857
Zhongang Qi
claimed_verified
2024-11-19T08:40:08.908Z
660cc64f1b14f691990c0ea0
/avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
Zhongang Qi
user
phoenixqza
66cbe22f8347e21e8a75a858
Ying Shan
66cbe22f8347e21e8a75a859
Xi Li
CustomCrafter: Customized Video Generation with Preserving Motion and
  Concept Composition Abilities
Customized video generation aims to generate high-quality videos guided by
text prompts and subject's reference images. However, since it is only trained
on static images, the fine-tuning process of subject learning disrupts
abilities of video diffusion models (VDMs) to combine concepts and generate
motions. To restore these abilities, some methods use additional video similar
to the prompt to fine-tune or guide the model. This requires frequent changes
of guiding videos and even re-tuning of the model when generating different
motions, which is very inconvenient for users. In this paper, we propose
CustomCrafter, a novel framework that preserves the model's motion generation
and conceptual combination abilities without additional video and fine-tuning
to recovery. For preserving conceptual combination ability, we design a
plug-and-play module to update few parameters in VDMs, enhancing the model's
ability to capture the appearance details and the ability of concept
combinations for new subjects. For motion generation, we observed that VDMs
tend to restore the motion of video in the early stage of denoising, while
focusing on the recovery of subject details in the later stage. Therefore, we
propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our
subject learning modules, we reduce the impact of this module on motion
generation in the early stage of denoising, preserving the ability to generate
motion of VDMs. In the later stage of denoising, we restore this module to
repair the appearance details of the specified subject, thereby ensuring the
fidelity of the subject's appearance. Experimental results show that our method
has a significant improvement compared to previous methods.
CustomCrafter enhances video diffusion models to preserve motion generation and concept combination without additional fine-tuning or guiding videos.
video diffusion models (VDMs) plug-and-play module Dynamic Weighted Video Sampling Strategy