MagicDance: Realistic Human Dance Video Generation with Motions & Facial
  Expressions Transfer
In this work, we propose MagicDance, a diffusion-based model for 2D human
motion and facial expression transfer on challenging human dance videos.
Specifically, we aim to generate human dance videos of any target identity
driven by novel pose sequences while keeping the identity unchanged. To this
end, we propose a two-stage training strategy to disentangle human motions and
appearance (e.g., facial expressions, skin tone and dressing), consisting of
the pretraining of an appearance-control block and fine-tuning of an
appearance-pose-joint-control block over human dance poses of the same dataset.
Our novel design enables robust appearance control with temporally consistent
upper body, facial attributes, and even background. The model also generalizes
well on unseen human identities and complex motion sequences without the need
for any fine-tuning with additional data with diverse human attributes by
leveraging the prior knowledge of image diffusion models. Moreover, the
proposed model is easy to use and can be considered as a plug-in
module/extension to Stable Diffusion. We also demonstrate the model's ability
for zero-shot 2D animation generation, enabling not only the appearance
transfer from one identity to another but also allowing for cartoon-like
stylization given only pose inputs. Extensive experiments demonstrate our
superior performance on the TikTok dataset.
655d825272ea9f3a3b829f20
Di Chang
admin_assigned
2023-11-22T13:00:30.654Z
64a5d8219f3b568c202b3137
https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d8219f3b568c202b3137/TI20Z1lWHzZpLsMkayDbU.png
Di Chang
user
Boese0601
655d825272ea9f3a3b829f21
Yichun Shi
admin_assigned
2023-11-22T13:00:41.562Z
6309ad81b105f8675bd5a796
/avatars/01c2186924486da4606e128e83709164.svg
Shi
user
Yichun
655d825272ea9f3a3b829f22
Quankai Gao
admin_assigned
2023-11-22T13:00:48.943Z
64c12bb1f149752445c39472
/avatars/8a534f5b850065f72dbe2e036181f7a9.svg
Quankai Gao
user
Quankai
655d825272ea9f3a3b829f23
Jessica Fu
admin_assigned
2023-11-22T13:00:56.246Z
64d042822f1f9578a0324917
/avatars/fea77fe4c09f38dfb649e445dc047370.svg
Jessica Fu
user
fujessc
655d825272ea9f3a3b829f24
Hongyi Xu
655d825272ea9f3a3b829f25
Guoxian Song
admin_assigned
2023-11-22T13:01:36.966Z
63086a237dc1b1a54cc6c24d
/avatars/477b94134edc4c18c8f769ecbb7d8091.svg
Song
user
guoxiansong
655d825272ea9f3a3b829f26
Qing Yan
655d825272ea9f3a3b829f27
Xiao Yang
claimed_verified
2024-07-03T07:42:33.352Z
62b3d2490385d6048985b42b
/avatars/1c4e534d086d53ef1057fbc00334350a.svg
Xiao Yang
user
seravee008
655d825272ea9f3a3b829f28
Mohammad Soleymani
MagicDance: Realistic Human Dance Video Generation with Motions & Facial
  Expressions Transfer
In this work, we propose MagicDance, a diffusion-based model for 2D human
motion and facial expression transfer on challenging human dance videos.
Specifically, we aim to generate human dance videos of any target identity
driven by novel pose sequences while keeping the identity unchanged. To this
end, we propose a two-stage training strategy to disentangle human motions and
appearance (e.g., facial expressions, skin tone and dressing), consisting of
the pretraining of an appearance-control block and fine-tuning of an
appearance-pose-joint-control block over human dance poses of the same dataset.
Our novel design enables robust appearance control with temporally consistent
upper body, facial attributes, and even background. The model also generalizes
well on unseen human identities and complex motion sequences without the need
for any fine-tuning with additional data with diverse human attributes by
leveraging the prior knowledge of image diffusion models. Moreover, the
proposed model is easy to use and can be considered as a plug-in
module/extension to Stable Diffusion. We also demonstrate the model's ability
for zero-shot 2D animation generation, enabling not only the appearance
transfer from one identity to another but also allowing for cartoon-like
stylization given only pose inputs. Extensive experiments demonstrate our
superior performance on the TikTok dataset.
MagicDance, a diffusion-based model, transfers human motion and facial expression across identities in dance videos, achieving high generalization and robust appearance control.
diffusion-based model 2D human motion facial expression transfer two-stage training strategy appearance-control block appearance-pose-joint-control block temporally consistent zero-shot 2D animation generation TikTok dataset