X-Dancer: Expressive Music to Human Dance Video Generation
We present X-Dancer, a novel zero-shot music-driven image animation pipeline
that creates diverse and long-range lifelike human dance videos from a single
static image. As its core, we introduce a unified transformer-diffusion
framework, featuring an autoregressive transformer model that synthesize
extended and music-synchronized token sequences for 2D body, head and hands
poses, which then guide a diffusion model to produce coherent and realistic
dance video frames. Unlike traditional methods that primarily generate human
motion in 3D, X-Dancer addresses data limitations and enhances scalability by
modeling a wide spectrum of 2D dance motions, capturing their nuanced alignment
with musical beats through readily available monocular videos. To achieve this,
we first build a spatially compositional token representation from 2D human
pose labels associated with keypoint confidences, encoding both large
articulated body movements (e.g., upper and lower body) and fine-grained
motions (e.g., head and hands). We then design a music-to-motion transformer
model that autoregressively generates music-aligned dance pose token sequences,
incorporating global attention to both musical style and prior motion context.
Finally we leverage a diffusion backbone to animate the reference image with
these synthesized pose tokens through AdaIN, forming a fully differentiable
end-to-end framework. Experimental results demonstrate that X-Dancer is able to
produce both diverse and characterized dance videos, substantially
outperforming state-of-the-art methods in term of diversity, expressiveness and
realism. Code and model will be available for research purposes.
67bd526001d5bfa0abfcc5ba
Zeyuan Chen
67bd526001d5bfa0abfcc5bb
Hongyi Xu
67bd526001d5bfa0abfcc5bc
Guoxian Song
admin_assigned
2025-02-25T16:33:13.695Z
63086a237dc1b1a54cc6c24d
/avatars/477b94134edc4c18c8f769ecbb7d8091.svg
Song
user
guoxiansong
67bd526001d5bfa0abfcc5bd
You Xie
admin_assigned
2025-02-25T16:33:21.430Z
6408dfd4b6a334f53e24023c
/avatars/b7e3fa4fbec6313e94ff3384b74dabfc.svg
You Xie
user
youxie
67bd526001d5bfa0abfcc5be
Chenxu Zhang
admin_assigned
2025-02-25T16:33:27.827Z
64f58e61abc51e6b0f885575
/avatars/53ec28d045e708570e0e34f44aaba7a7.svg
Chenxu Zhang
user
ChenxuZhang528
67bd526001d5bfa0abfcc5bf
Xin Chen
67bd526001d5bfa0abfcc5c0
Chao Wang
admin_assigned
2025-02-25T16:33:39.754Z
64cf8caa0b71aea8be5c97db
/avatars/d486466afa3b1d58abd85725930b9298.svg
Chao Wang
user
chaowang
67bd526001d5bfa0abfcc5c1
Di Chang
admin_assigned
2025-02-25T16:33:46.334Z
64a5d8219f3b568c202b3137
https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d8219f3b568c202b3137/TI20Z1lWHzZpLsMkayDbU.png
Di Chang
user
Boese0601
67bd526001d5bfa0abfcc5c2
Linjie Luo
X-Dancer: Expressive Music to Human Dance Video Generation
We present X-Dancer, a novel zero-shot music-driven image animation pipeline
that creates diverse and long-range lifelike human dance videos from a single
static image. As its core, we introduce a unified transformer-diffusion
framework, featuring an autoregressive transformer model that synthesize
extended and music-synchronized token sequences for 2D body, head and hands
poses, which then guide a diffusion model to produce coherent and realistic
dance video frames. Unlike traditional methods that primarily generate human
motion in 3D, X-Dancer addresses data limitations and enhances scalability by
modeling a wide spectrum of 2D dance motions, capturing their nuanced alignment
with musical beats through readily available monocular videos. To achieve this,
we first build a spatially compositional token representation from 2D human
pose labels associated with keypoint confidences, encoding both large
articulated body movements (e.g., upper and lower body) and fine-grained
motions (e.g., head and hands). We then design a music-to-motion transformer
model that autoregressively generates music-aligned dance pose token sequences,
incorporating global attention to both musical style and prior motion context.
Finally we leverage a diffusion backbone to animate the reference image with
these synthesized pose tokens through AdaIN, forming a fully differentiable
end-to-end framework. Experimental results demonstrate that X-Dancer is able to
produce both diverse and characterized dance videos, substantially
outperforming state-of-the-art methods in term of diversity, expressiveness and
realism. Code and model will be available for research purposes.
A zero-shot music-driven image animation system generates diverse and realistic dance videos from a single static image using a unified transformer-diffusion framework with autoregressive pose generation and music synchronization.
transformer diffusion model autoregressive token sequences 2D poses diffusion backbone AdaIN music-to-motion global attention