Audio Conditioning for Music Generation via Discrete Bottleneck Features
While most music generation models use textual or parametric conditioning
(e.g. tempo, harmony, musical genre), we propose to condition a language model
based music generation system with audio input. Our exploration involves two
distinct strategies. The first strategy, termed textual inversion, leverages a
pre-trained text-to-music model to map audio input to corresponding
"pseudowords" in the textual embedding space. For the second model we train a
music language model from scratch jointly with a text conditioner and a
quantized audio feature extractor. At inference time, we can mix textual and
audio conditioning and balance them thanks to a novel double classifier free
guidance method. We conduct automatic and human studies that validates our
approach. We will release the code and we provide music samples on
https://musicgenstyle.github.io in order to show the quality of our model.
66987bdba9778d20ff251561
Simon Rouard
extracted_confirmed
2024-10-03T11:58:08.371Z
66fe867df44983580b27700d
/avatars/d2af73b02e1b361dd9e25dcc83c890a0.svg
Simon Rouard
user
simonrouard
66987bdba9778d20ff251562
Yossi Adi
66987bdba9778d20ff251563
Jade Copet
66987bdba9778d20ff251564
Axel Roebel
66987bdba9778d20ff251565
Alexandre Défossez
Audio Conditioning for Music Generation via Discrete Bottleneck Features
While most music generation models use textual or parametric conditioning
(e.g. tempo, harmony, musical genre), we propose to condition a language model
based music generation system with audio input. Our exploration involves two
distinct strategies. The first strategy, termed textual inversion, leverages a
pre-trained text-to-music model to map audio input to corresponding
"pseudowords" in the textual embedding space. For the second model we train a
music language model from scratch jointly with a text conditioner and a
quantized audio feature extractor. At inference time, we can mix textual and
audio conditioning and balance them thanks to a novel double classifier free
guidance method. We conduct automatic and human studies that validates our
approach. We will release the code and we provide music samples on
https://musicgenstyle.github.io in order to show the quality of our model.
A music generation system uses audio input for conditioning, employing both textual inversion and a novel joint training approach with text and audio features, achieving high-quality results.
text-to-music model pseudowords textual embedding space music language model quantized audio feature extractor double classifier-free guidance method