DisCo: Disentangled Control for Referring Human Dance Generation in Real
  World
Generative AI has made significant strides in computer vision, particularly
in image/video synthesis conditioned on text descriptions. Despite the
advancements, it remains challenging especially in the generation of
human-centric content such as dance synthesis. Existing dance synthesis methods
struggle with the gap between synthesized content and real-world dance
scenarios. In this paper, we define a new problem setting: Referring Human
Dance Generation, which focuses on real-world dance scenarios with three
important properties: (i) Faithfulness: the synthesis should retain the
appearance of both human subject foreground and background from the reference
image, and precisely follow the target pose; (ii) Generalizability: the model
should generalize to unseen human subjects, backgrounds, and poses; (iii)
Compositionality: it should allow for composition of seen/unseen subjects,
backgrounds, and poses from different sources. To address these challenges, we
introduce a novel approach, DISCO, which includes a novel model architecture
with disentangled control to improve the faithfulness and compositionality of
dance synthesis, and an effective human attribute pre-training for better
generalizability to unseen humans. Extensive qualitative and quantitative
results demonstrate that DISCO can generate high-quality human dance images and
videos with diverse appearances and flexible motions. Code, demo, video and
visualization are available at: https://disco-dance.github.io/.
64a39ecbc9b4db7cc77e31c0
Tan Wang
claimed_verified
2023-07-04T19:16:38.476Z
62a9cb5c0036766ec1b67f09
/avatars/5a9c5b41ef3eebbe7b21184b289ade68.svg
Tan Wang
user
ReaLMe
64a39ecbc9b4db7cc77e31c1
Linjie Li
claimed_verified
2023-08-08T01:20:23.803Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
64a39ecbc9b4db7cc77e31c2
Kevin Lin
claimed_verified
2023-07-05T18:56:05.428Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
64a39ecbc9b4db7cc77e31c3
Chung-Ching Lin
claimed_verified
2024-06-13T09:33:59.307Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
64a39ecbc9b4db7cc77e31c4
Zhengyuan Yang
64a39ecbc9b4db7cc77e31c5
Hanwang Zhang
64a39ecbc9b4db7cc77e31c6
Zicheng Liu
64a39ecbc9b4db7cc77e31c7
Lijuan Wang
DisCo: Disentangled Control for Referring Human Dance Generation in Real
  World
Generative AI has made significant strides in computer vision, particularly
in image/video synthesis conditioned on text descriptions. Despite the
advancements, it remains challenging especially in the generation of
human-centric content such as dance synthesis. Existing dance synthesis methods
struggle with the gap between synthesized content and real-world dance
scenarios. In this paper, we define a new problem setting: Referring Human
Dance Generation, which focuses on real-world dance scenarios with three
important properties: (i) Faithfulness: the synthesis should retain the
appearance of both human subject foreground and background from the reference
image, and precisely follow the target pose; (ii) Generalizability: the model
should generalize to unseen human subjects, backgrounds, and poses; (iii)
Compositionality: it should allow for composition of seen/unseen subjects,
backgrounds, and poses from different sources. To address these challenges, we
introduce a novel approach, DISCO, which includes a novel model architecture
with disentangled control to improve the faithfulness and compositionality of
dance synthesis, and an effective human attribute pre-training for better
generalizability to unseen humans. Extensive qualitative and quantitative
results demonstrate that DISCO can generate high-quality human dance images and
videos with diverse appearances and flexible motions. Code, demo, video and
visualization are available at: https://disco-dance.github.io/.
DISCO addresses the challenges of human-centric dance synthesis by introducing a model architecture for better faithfulness and compositionality, and pre-training for generalizability.
disentangled control Referring Human Dance Generation