MusicLDM: Enhancing Novelty in Text-to-Music Generation Using
  Beat-Synchronous Mixup Strategies
Diffusion models have shown promising results in cross-modal generation
tasks, including text-to-image and text-to-audio generation. However,
generating music, as a special type of audio, presents unique challenges due to
limited availability of music data and sensitive issues related to copyright
and plagiarism. In this paper, to tackle these challenges, we first construct a
state-of-the-art text-to-music model, MusicLDM, that adapts Stable Diffusion
and AudioLDM architectures to the music domain. We achieve this by retraining
the contrastive language-audio pretraining model (CLAP) and the Hifi-GAN
vocoder, as components of MusicLDM, on a collection of music data samples.
Then, to address the limitations of training data and to avoid plagiarism, we
leverage a beat tracking model and propose two different mixup strategies for
data augmentation: beat-synchronous audio mixup and beat-synchronous latent
mixup, which recombine training audio directly or via a latent embeddings
space, respectively. Such mixup strategies encourage the model to interpolate
between musical training samples and generate new music within the convex hull
of the training data, making the generated music more diverse while still
staying faithful to the corresponding style. In addition to popular evaluation
metrics, we design several new evaluation metrics based on CLAP score to
demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies
improve both the quality and novelty of generated music, as well as the
correspondence between input text and generated music.
64cc5a70eed22517ae0d4db1
Ke Chen
claimed_verified
2023-09-17T19:21:44.450Z
63fe457052441fe3e874ee22
https://cdn-avatars.huggingface.co/v1/production/uploads/1677608254704-noauth.jpeg
Ke Chen
user
RSNuts
64cc5a70eed22517ae0d4db2
Yusong Wu
admin_assigned
2023-08-04T12:45:31.178Z
61c3239277d54877f764328e
/avatars/2d2894cb3d61942cf8661c78350f3c3e.svg
Yusong Wu
user
lukewys
64cc5a70eed22517ae0d4db3
Haohe Liu
admin_assigned
2023-08-04T12:45:14.771Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
64cc5a70eed22517ae0d4db4
Marianna Nezhurina
admin_assigned
2023-08-04T12:44:52.422Z
62500f361684e0335e527bc6
/avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
Marianna Nezhurina
user
marianna13
64cc5a70eed22517ae0d4db5
Taylor Berg-Kirkpatrick
64cc5a70eed22517ae0d4db6
Shlomo Dubnov
claimed_verified
2024-03-13T16:15:20.366Z
643da2c837ca9134b589099a
/avatars/67c317d2b1644b127c4eed53da301ad9.svg
Shlomo Dubnov
user
sdubnov
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using
  Beat-Synchronous Mixup Strategies
Diffusion models have shown promising results in cross-modal generation
tasks, including text-to-image and text-to-audio generation. However,
generating music, as a special type of audio, presents unique challenges due to
limited availability of music data and sensitive issues related to copyright
and plagiarism. In this paper, to tackle these challenges, we first construct a
state-of-the-art text-to-music model, MusicLDM, that adapts Stable Diffusion
and AudioLDM architectures to the music domain. We achieve this by retraining
the contrastive language-audio pretraining model (CLAP) and the Hifi-GAN
vocoder, as components of MusicLDM, on a collection of music data samples.
Then, to address the limitations of training data and to avoid plagiarism, we
leverage a beat tracking model and propose two different mixup strategies for
data augmentation: beat-synchronous audio mixup and beat-synchronous latent
mixup, which recombine training audio directly or via a latent embeddings
space, respectively. Such mixup strategies encourage the model to interpolate
between musical training samples and generate new music within the convex hull
of the training data, making the generated music more diverse while still
staying faithful to the corresponding style. In addition to popular evaluation
metrics, we design several new evaluation metrics based on CLAP score to
demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies
improve both the quality and novelty of generated music, as well as the
correspondence between input text and generated music.
A text-to-music model named MusicLDM, based on Stable Diffusion and AudioLDM, overcomes limitations in music data and plagiarism with beat tracking and mixup strategies, enhancing music quality, novelty, and text correspondence.
diffusion models text-to-music MusicLDM Stable Diffusion AudioLDM contrastive language-audio pretraining model CLAP Hifi-GAN beat tracking beat-synchronous audio mixup beat-synchronous latent mixup CLAP score