DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
  Motion Control
Recent advances in customized video generation have enabled users to create
videos tailored to both specific subjects and motion trajectories. However,
existing methods often require complicated test-time fine-tuning and struggle
with balancing subject learning and motion control, limiting their real-world
applications. In this paper, we present DreamVideo-2, a zero-shot video
customization framework capable of generating videos with a specific subject
and motion trajectory, guided by a single image and a bounding box sequence,
respectively, and without the need for test-time fine-tuning. Specifically, we
introduce reference attention, which leverages the model's inherent
capabilities for subject learning, and devise a mask-guided motion module to
achieve precise motion control by fully utilizing the robust motion signal of
box masks derived from bounding boxes. While these two components achieve their
intended functions, we empirically observe that motion control tends to
dominate over subject learning. To address this, we propose two key designs: 1)
the masked reference attention, which integrates a blended latent mask modeling
scheme into reference attention to enhance subject representations at the
desired positions, and 2) a reweighted diffusion loss, which differentiates the
contributions of regions inside and outside the bounding boxes to ensure a
balance between subject and motion control. Extensive experimental results on a
newly curated dataset demonstrate that DreamVideo-2 outperforms
state-of-the-art methods in both subject customization and motion control. The
dataset, code, and models will be made publicly available.
6711c8f0fe58fd40dde8c60a
Yujie Wei
admin_assigned
2024-10-18T12:42:53.127Z
637f70d6fab5db9101c3dfc8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
Yujie Wei
user
weilllllls
6711c8f0fe58fd40dde8c60b
Shiwei Zhang
6711c8f0fe58fd40dde8c60c
Hangjie Yuan
admin_assigned
2024-10-18T12:53:28.821Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
6711c8f0fe58fd40dde8c60d
Xiang Wang
6711c8f0fe58fd40dde8c60e
Haonan Qiu
admin_assigned
2024-10-18T12:52:59.555Z
63023b6ab002e9a4a2152890
/avatars/cae8ba0a8d61fb4e576934431f43991b.svg
Haonan Qiu
user
MoonQiu
6711c8f0fe58fd40dde8c60f
Rui Zhao
6711c8f0fe58fd40dde8c610
Yutong Feng
admin_assigned
2024-10-18T12:53:05.174Z
64a54e468cfaa458bd6844bf
https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
Yutong Feng
user
fengyutong
6711c8f0fe58fd40dde8c611
Feng Liu
claimed_verified
2024-12-02T09:41:00.897Z
661658b9e398e4f0757221ce
/avatars/9e9d55145661d00e2a9e029ba2e085d8.svg
Liu Feng
user
LiewFeng
6711c8f0fe58fd40dde8c612
Zhizhong Huang
admin_assigned
2024-10-18T12:53:11.406Z
64c4aad5d68946edad616413
https://cdn-avatars.huggingface.co/v1/production/uploads/64c4aad5d68946edad616413/RY2lSiwOP5tDKDQWBF7E1.png
Zhizhong Huang
user
Hzzone
6711c8f0fe58fd40dde8c613
Jiaxin Ye
admin_assigned
2024-10-18T12:43:21.961Z
6503b02993574a89717b8f91
/avatars/f4d1698f1f7c516c5b849b96351a885e.svg
JiaxinYe
user
JiaxinYe
6711c8f0fe58fd40dde8c614
Yingya Zhang
6711c8f0fe58fd40dde8c615
Hongming Shan
DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
  Motion Control
Recent advances in customized video generation have enabled users to create
videos tailored to both specific subjects and motion trajectories. However,
existing methods often require complicated test-time fine-tuning and struggle
with balancing subject learning and motion control, limiting their real-world
applications. In this paper, we present DreamVideo-2, a zero-shot video
customization framework capable of generating videos with a specific subject
and motion trajectory, guided by a single image and a bounding box sequence,
respectively, and without the need for test-time fine-tuning. Specifically, we
introduce reference attention, which leverages the model's inherent
capabilities for subject learning, and devise a mask-guided motion module to
achieve precise motion control by fully utilizing the robust motion signal of
box masks derived from bounding boxes. While these two components achieve their
intended functions, we empirically observe that motion control tends to
dominate over subject learning. To address this, we propose two key designs: 1)
the masked reference attention, which integrates a blended latent mask modeling
scheme into reference attention to enhance subject representations at the
desired positions, and 2) a reweighted diffusion loss, which differentiates the
contributions of regions inside and outside the bounding boxes to ensure a
balance between subject and motion control. Extensive experimental results on a
newly curated dataset demonstrate that DreamVideo-2 outperforms
state-of-the-art methods in both subject customization and motion control. The
dataset, code, and models will be made publicly available.
DreamVideo-2 generates customized videos using a single image and bounding box sequence without test-time fine-tuning, balancing subject learning and motion control through masked reference attention and reweighted diffusion loss.
zero-shot video customization reference attention mask-guided motion module blended latent mask modeling reweighted diffusion loss