Versatile Framework for Song Generation with Prompt-based Control
Song generation focuses on producing controllable high-quality songs based on
various prompts. However, existing methods struggle to generate vocals and
accompaniments with prompt-based control and proper alignment. Additionally,
they fall short in supporting various tasks. To address these challenges, we
introduce VersBand, a multi-task song generation framework for synthesizing
high-quality, aligned songs with prompt-based control. VersBand comprises these
primary models: 1) VocalBand, a decoupled model, leverages the flow-matching
method for generating singing styles, pitches, and mel-spectrograms, allowing
fast, high-quality vocal generation with style control. 2) AccompBand, a
flow-based transformer model, incorporates the Band-MOE, selecting suitable
experts for enhanced quality, alignment, and control. This model allows for
generating controllable, high-quality accompaniments aligned with vocals. 3)
Two generation models, LyricBand for lyrics and MelodyBand for melodies,
contribute to the comprehensive multi-task song generation system, allowing for
extensive control based on multiple prompts. Experimental results demonstrate
that VersBand performs better over baseline models across multiple song
generation tasks using objective and subjective metrics. Audio samples are
available at https://VersBand.github.io.
6810d9d76d704ff9ea87ab82
Yu Zhang
claimed_verified
2025-04-30T09:56:44.960Z
66569729ea21cfae5f5797c4
https://cdn-avatars.huggingface.co/v1/production/uploads/66569729ea21cfae5f5797c4/IguwJzljFN3QiEd1bn5BP.jpeg
Yu Zhang
user
AaronZ345
6810d9d76d704ff9ea87ab83
Wenxiang Guo
6810d9d76d704ff9ea87ab84
Changhao Pan
6810d9d76d704ff9ea87ab85
Zhiyuan Zhu
6810d9d76d704ff9ea87ab86
Ruiqi Li
6810d9d76d704ff9ea87ab87
Jingyu Lu
6810d9d76d704ff9ea87ab88
Rongjie Huang
6810d9d76d704ff9ea87ab89
Ruiyuan Zhang
6810d9d76d704ff9ea87ab8a
Zhiqing Hong
6810d9d76d704ff9ea87ab8b
Ziyue Jiang
6810d9d76d704ff9ea87ab8c
Zhou Zhao
Versatile Framework for Song Generation with Prompt-based Control
Song generation focuses on producing controllable high-quality songs based on
various prompts. However, existing methods struggle to generate vocals and
accompaniments with prompt-based control and proper alignment. Additionally,
they fall short in supporting various tasks. To address these challenges, we
introduce VersBand, a multi-task song generation framework for synthesizing
high-quality, aligned songs with prompt-based control. VersBand comprises these
primary models: 1) VocalBand, a decoupled model, leverages the flow-matching
method for generating singing styles, pitches, and mel-spectrograms, allowing
fast, high-quality vocal generation with style control. 2) AccompBand, a
flow-based transformer model, incorporates the Band-MOE, selecting suitable
experts for enhanced quality, alignment, and control. This model allows for
generating controllable, high-quality accompaniments aligned with vocals. 3)
Two generation models, LyricBand for lyrics and MelodyBand for melodies,
contribute to the comprehensive multi-task song generation system, allowing for
extensive control based on multiple prompts. Experimental results demonstrate
that VersBand performs better over baseline models across multiple song
generation tasks using objective and subjective metrics. Audio samples are
available at https://VersBand.github.io.
VersBand is a multi-task song generation framework that integrates multiple models for high-quality, aligned vocals and accompaniments with prompt-based control.
flow-matching flow-based transformer Band-MOE multi-task song generation vocal generation accompaniment generation lyrics generation melody generation