UniMuMo: Unified Text, Music and Motion Generation
We introduce UniMuMo, a unified multimodal model capable of taking arbitrary
text, music, and motion data as input conditions to generate outputs across all
three modalities. To address the lack of time-synchronized data, we align
unpaired music and motion data based on rhythmic patterns to leverage existing
large-scale music-only and motion-only datasets. By converting music, motion,
and text into token-based representation, our model bridges these modalities
through a unified encoder-decoder transformer architecture. To support multiple
generation tasks within a single framework, we introduce several architectural
improvements. We propose encoding motion with a music codebook, mapping motion
into the same feature space as music. We introduce a music-motion parallel
generation scheme that unifies all music and motion generation tasks into a
single transformer decoder architecture with a single training task of
music-motion joint generation. Moreover, the model is designed by fine-tuning
existing pre-trained single-modality models, significantly reducing
computational demands. Extensive experiments demonstrate that UniMuMo achieves
competitive results on all unidirectional generation benchmarks across music,
motion, and text modalities. Quantitative results are available in the
https://hanyangclarence.github.io/unimumo_demo/{project page}.
6704b563755b74099845d572
Han Yang
claimed_verified
2024-10-08T12:46:20.473Z
63ef2578436d28737ee8925c
/avatars/cf854d419673d55a6822b7e156dade88.svg
YH
user
ClarenceY
6704b563755b74099845d573
Kun Su
6704b563755b74099845d574
Yutong Zhang
6704b563755b74099845d575
Jiaben Chen
admin_assigned
2024-10-08T09:57:34.574Z
64f788ea9980b96c33e2b8fe
/avatars/b9fc9a93550df79033f86758fc8cbc5d.svg
Jiaben Chen
user
Jiaben
6704b563755b74099845d576
Kaizhi Qian
admin_assigned
2024-10-08T09:57:28.773Z
664a316b658c482f96eba4c0
/avatars/0447a713eec179e423613b33e668bf68.svg
Kaizhi Qian
user
auspicious3000
6704b563755b74099845d577
Gaowen Liu
6704b563755b74099845d578
Chuang Gan
UniMuMo: Unified Text, Music and Motion Generation
We introduce UniMuMo, a unified multimodal model capable of taking arbitrary
text, music, and motion data as input conditions to generate outputs across all
three modalities. To address the lack of time-synchronized data, we align
unpaired music and motion data based on rhythmic patterns to leverage existing
large-scale music-only and motion-only datasets. By converting music, motion,
and text into token-based representation, our model bridges these modalities
through a unified encoder-decoder transformer architecture. To support multiple
generation tasks within a single framework, we introduce several architectural
improvements. We propose encoding motion with a music codebook, mapping motion
into the same feature space as music. We introduce a music-motion parallel
generation scheme that unifies all music and motion generation tasks into a
single transformer decoder architecture with a single training task of
music-motion joint generation. Moreover, the model is designed by fine-tuning
existing pre-trained single-modality models, significantly reducing
computational demands. Extensive experiments demonstrate that UniMuMo achieves
competitive results on all unidirectional generation benchmarks across music,
motion, and text modalities. Quantitative results are available in the
https://hanyangclarence.github.io/unimumo_demo/{project page}.
UniMuMo generates outputs across text, music, and motion modalities using a unified encoder-decoder transformer architecture and aligned unpaired data based on rhythmic patterns.
Unified multimodal model token-based representation encoder-decoder transformer architecture music codebook music-motion parallel generation single transformer decoder architecture music-motion joint generation pre-trained single-modality models