M^{2}UGen: Multi-modal Music Understanding and Generation with the
  Power of Large Language Models
The current landscape of research leveraging large language models (LLMs) is
experiencing a surge. Many works harness the powerful reasoning capabilities of
these models to comprehend various modalities, such as text, speech, images,
videos, etc. They also utilize LLMs to understand human intention and generate
desired outputs like images, videos, and music. However, research that combines
both understanding and generation using LLMs is still limited and in its
nascent stage. To address this gap, we introduce a Multi-modal Music
Understanding and Generation (M^{2}UGen) framework that integrates LLM's
abilities to comprehend and generate music for different modalities. The
M^{2}UGen framework is purpose-built to unlock creative potential from
diverse sources of inspiration, encompassing music, image, and video through
the use of pretrained MERT, ViT, and ViViT models, respectively. To enable
music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging
multi-modal understanding and music generation is accomplished through the
integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
model to generate extensive datasets that support text/image/video-to-music
generation, facilitating the training of our M^{2}UGen framework. We conduct
a thorough evaluation of our proposed framework. The experimental results
demonstrate that our model achieves or surpasses the performance of the current
state-of-the-art models.
655c295c973f30a0a593ed04
Atin Sakkeer Hussain
admin_assigned
2023-11-21T14:46:27.677Z
61b4db3d8c3a7fcce3c71a25
https://cdn-avatars.huggingface.co/v1/production/uploads/61b4db3d8c3a7fcce3c71a25/7I035TZlmlTFHtsut92rG.jpeg
Atin Sakkeer Hussain
user
crypto-code
655c295c973f30a0a593ed05
Shansong Liu
admin_assigned
2023-11-21T16:38:20.422Z
64c1f3560a2545084f08350b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c1f3560a2545084f08350b/RbSzN5eFvExrfRLX1Xi_g.jpeg
Shansong Liu
user
dadinghh2
655c295c973f30a0a593ed06
Chenshuo Sun
655c295c973f30a0a593ed07
Ying Shan
M^{2}UGen: Multi-modal Music Understanding and Generation with the
  Power of Large Language Models
The current landscape of research leveraging large language models (LLMs) is
experiencing a surge. Many works harness the powerful reasoning capabilities of
these models to comprehend various modalities, such as text, speech, images,
videos, etc. They also utilize LLMs to understand human intention and generate
desired outputs like images, videos, and music. However, research that combines
both understanding and generation using LLMs is still limited and in its
nascent stage. To address this gap, we introduce a Multi-modal Music
Understanding and Generation (M^{2}UGen) framework that integrates LLM's
abilities to comprehend and generate music for different modalities. The
M^{2}UGen framework is purpose-built to unlock creative potential from
diverse sources of inspiration, encompassing music, image, and video through
the use of pretrained MERT, ViT, and ViViT models, respectively. To enable
music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging
multi-modal understanding and music generation is accomplished through the
integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
model to generate extensive datasets that support text/image/video-to-music
generation, facilitating the training of our M^{2}UGen framework. We conduct
a thorough evaluation of our proposed framework. The experimental results
demonstrate that our model achieves or surpasses the performance of the current
state-of-the-art models.
A framework integrating large language models for multi-modal music understanding and generation outperforms current state-of-the-art models.
Multi-modal Music Understanding and Generation M$^{2}$UGen pretrained MERT ViT ViViT AudioLDM 2 MusicGen LLaMA 2 MU-LLaMA text/image/video-to-music generation