JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
  Parameters Tuning
Large models for text-to-music generation have achieved significant progress,
facilitating the creation of high-quality and varied musical compositions from
provided text prompts. However, input text prompts may not precisely capture
user requirements, particularly when the objective is to generate music that
embodies a specific concept derived from a designated reference collection. In
this paper, we propose a novel method for customized text-to-music generation,
which can capture the concept from a two-minute reference music and generate a
new piece of music conforming to the concept. We achieve this by fine-tuning a
pretrained text-to-music model using the reference music. However, directly
fine-tuning all parameters leads to overfitting issues. To address this
problem, we propose a Pivotal Parameters Tuning method that enables the model
to assimilate the new concept while preserving its original generative
capabilities. Additionally, we identify a potential concept conflict when
introducing multiple concepts into the pretrained model. We present a concept
enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
model to generate music incorporating either individual or multiple concepts
simultaneously. Since we are the first to work on the customized music
generation task, we also introduce a new dataset and evaluation protocol for
the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
both qualitative and quantitative evaluations. Demos will be available at
https://www.jenmusic.ai/research#DreamStyler.
6672597e7e7ff5d8bd0b18cf
Boyu Chen
6672597e7e7ff5d8bd0b18d0
Peike Li
admin_assigned
2024-06-19T21:11:00.982Z
6367054f361a96184dc44db0
https://cdn-avatars.huggingface.co/v1/production/uploads/1667695944853-noauth.png
Li
user
Peike
6672597e7e7ff5d8bd0b18d1
Yao Yao
6672597e7e7ff5d8bd0b18d2
Alex Wang
claimed_verified
2024-06-20T07:34:27.805Z
6673cbc4ccd71b5bb7609a62
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/M543BjPm3K4A-OBYOFUEE.png
Dr. Alex Wang
user
dralexw
JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
  Parameters Tuning
Large models for text-to-music generation have achieved significant progress,
facilitating the creation of high-quality and varied musical compositions from
provided text prompts. However, input text prompts may not precisely capture
user requirements, particularly when the objective is to generate music that
embodies a specific concept derived from a designated reference collection. In
this paper, we propose a novel method for customized text-to-music generation,
which can capture the concept from a two-minute reference music and generate a
new piece of music conforming to the concept. We achieve this by fine-tuning a
pretrained text-to-music model using the reference music. However, directly
fine-tuning all parameters leads to overfitting issues. To address this
problem, we propose a Pivotal Parameters Tuning method that enables the model
to assimilate the new concept while preserving its original generative
capabilities. Additionally, we identify a potential concept conflict when
introducing multiple concepts into the pretrained model. We present a concept
enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
model to generate music incorporating either individual or multiple concepts
simultaneously. Since we are the first to work on the customized music
generation task, we also introduce a new dataset and evaluation protocol for
the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
both qualitative and quantitative evaluations. Demos will be available at
https://www.jenmusic.ai/research#DreamStyler.
A novel method for customized text-to-music generation using pivotal parameters tuning and concept enhancement outperforms baselines in qualitative and quantitative evaluations.
text-to-music generation pretrained model fine-tuning overfitting pivotal parameters tuning concept conflict concept enhancement new dataset evaluation protocol Jen1-DreamStyler