MixerMDM: Learnable Composition of Human Motion Diffusion Models
Generating human motion guided by conditions such as textual descriptions is
challenging due to the need for datasets with pairs of high-quality motion and
their corresponding conditions. The difficulty increases when aiming for finer
control in the generation. To that end, prior works have proposed to combine
several motion diffusion models pre-trained on datasets with different types of
conditions, thus allowing control with multiple conditions. However, the
proposed merging strategies overlook that the optimal way to combine the
generation processes might depend on the particularities of each pre-trained
generative model and also the specific textual descriptions. In this context,
we introduce MixerMDM, the first learnable model composition technique for
combining pre-trained text-conditioned human motion diffusion models. Unlike
previous approaches, MixerMDM provides a dynamic mixing strategy that is
trained in an adversarial fashion to learn to combine the denoising process of
each model depending on the set of conditions driving the generation. By using
MixerMDM to combine single- and multi-person motion diffusion models, we
achieve fine-grained control on the dynamics of every person individually, and
also on the overall interaction. Furthermore, we propose a new evaluation
technique that, for the first time in this task, measures the interaction and
individual quality by computing the alignment between the mixed generated
motions and their conditions as well as the capabilities of MixerMDM to adapt
the mixing throughout the denoising process depending on the motions to mix.
67ecd893553eb1c01a311882
Pablo Ruiz-Ponce
claimed_verified
2025-04-02T08:21:16.654Z
62ace9bc717ee4c12b72e275
https://cdn-avatars.huggingface.co/v1/production/uploads/62ace9bc717ee4c12b72e275/VfR-7R3H-5FV0OILa0XzL.jpeg
Pablo Ruiz-Ponce
user
pabloruizponce
67ecd893553eb1c01a311883
German Barquero
admin_assigned
2025-04-02T09:53:49.237Z
64fac0f3b961d0d12c756b59
/avatars/27cc7dbad927df818ba2f91a5b6942f9.svg
German Barquero
user
Germs96
67ecd893553eb1c01a311884
Cristina Palmero
67ecd893553eb1c01a311885
Sergio Escalera
67ecd893553eb1c01a311886
José García-Rodríguez
MixerMDM: Learnable Composition of Human Motion Diffusion Models
Generating human motion guided by conditions such as textual descriptions is
challenging due to the need for datasets with pairs of high-quality motion and
their corresponding conditions. The difficulty increases when aiming for finer
control in the generation. To that end, prior works have proposed to combine
several motion diffusion models pre-trained on datasets with different types of
conditions, thus allowing control with multiple conditions. However, the
proposed merging strategies overlook that the optimal way to combine the
generation processes might depend on the particularities of each pre-trained
generative model and also the specific textual descriptions. In this context,
we introduce MixerMDM, the first learnable model composition technique for
combining pre-trained text-conditioned human motion diffusion models. Unlike
previous approaches, MixerMDM provides a dynamic mixing strategy that is
trained in an adversarial fashion to learn to combine the denoising process of
each model depending on the set of conditions driving the generation. By using
MixerMDM to combine single- and multi-person motion diffusion models, we
achieve fine-grained control on the dynamics of every person individually, and
also on the overall interaction. Furthermore, we propose a new evaluation
technique that, for the first time in this task, measures the interaction and
individual quality by computing the alignment between the mixed generated
motions and their conditions as well as the capabilities of MixerMDM to adapt
the mixing throughout the denoising process depending on the motions to mix.
MixerMDM, a learnable technique, dynamically combines pre-trained text-conditioned motion diffusion models to achieve fine-grained control over individual and interactive human motions, using an adversarial training approach and new evaluation metrics.
motion diffusion models text-conditioned learnable model composition dynamic mixing adversarial fashion denoising process