MusicRL: Aligning Music Generation to Human Preferences
We propose MusicRL, the first music generation system finetuned from human
feedback. Appreciation of text-to-music models is particularly subjective since
the concept of musicality as well as the specific intention behind a caption
are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
retro guitar solo or a techno pop beat). Not only this makes supervised
training of such models challenging, but it also calls for integrating
continuous human feedback in their post-deployment finetuning. MusicRL is a
pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
audio tokens finetuned with reinforcement learning to maximise sequence-level
rewards. We design reward functions related specifically to text-adherence and
audio quality with the help from selected raters, and use those to finetune
MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
that incorporates human feedback at scale. Human evaluations show that both
MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
combines the two approaches and results in the best model according to human
raters. Ablation studies shed light on the musical attributes influencing human
preferences, indicating that text adherence and quality only account for a part
of it. This underscores the prevalence of subjectivity in musical appreciation
and calls for further involvement of human listeners in the finetuning of music
generation models.
65c30546047c54295aba2855
Geoffrey Cideron
65c30546047c54295aba2856
Sertan Girgin
65c30546047c54295aba2857
Mauro Verzetti
65c30546047c54295aba2858
Damien Vincent
65c30546047c54295aba2859
Matej Kastelic
65c30546047c54295aba285a
Zalán Borsos
65c30546047c54295aba285b
Brian McWilliams
65c30546047c54295aba285c
Victor Ungureanu
65c30546047c54295aba285d
Olivier Bachem
65c30546047c54295aba285e
Olivier Pietquin
65c30546047c54295aba285f
Matthieu Geist
claimed_verified
2025-05-19T13:09:42.426Z
67bef153eaca0bef78ed5478
/avatars/ab838c541bd170986370a99670981c2f.svg
Matthieu Geist
user
matthieu-geist
65c30546047c54295aba2860
Léonard Hussenot
claimed_verified
2025-03-28T14:22:44.850Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
65c30546047c54295aba2861
Neil Zeghidour
65c30546047c54295aba2862
Andrea Agostinelli
MusicRL: Aligning Music Generation to Human Preferences
We propose MusicRL, the first music generation system finetuned from human
feedback. Appreciation of text-to-music models is particularly subjective since
the concept of musicality as well as the specific intention behind a caption
are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
retro guitar solo or a techno pop beat). Not only this makes supervised
training of such models challenging, but it also calls for integrating
continuous human feedback in their post-deployment finetuning. MusicRL is a
pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
audio tokens finetuned with reinforcement learning to maximise sequence-level
rewards. We design reward functions related specifically to text-adherence and
audio quality with the help from selected raters, and use those to finetune
MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
that incorporates human feedback at scale. Human evaluations show that both
MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
combines the two approaches and results in the best model according to human
raters. Ablation studies shed light on the musical attributes influencing human
preferences, indicating that text adherence and quality only account for a part
of it. This underscores the prevalence of subjectivity in musical appreciation
and calls for further involvement of human listeners in the finetuning of music
generation models.
MusicRL, a pretrained MusicLM model refined using reinforcement learning from human feedback, produces high-quality text-to-music output preferred by human raters over a baseline model.
MusicRL MusicLM autoregressive discrete audio tokens reinforcement learning sequence-level rewards reward functions text-adherence audio quality Reinforcement Learning from Human Feedback (RLHF) ablation studies