FLUX that Plays Music
This paper explores a simple extension of diffusion-based rectified flow
Transformers for text-to-music generation, termed as FluxMusic. Generally,
along with design in advanced
Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
into a latent VAE space of mel-spectrum. It involves first applying a sequence
of independent attention to the double text-music stream, followed by a stacked
single music stream for denoised patch prediction. We employ multiple
pre-trained text encoders to sufficiently capture caption semantic information
as well as inference flexibility. In between, coarse textual information, in
conjunction with time step embeddings, is utilized in a modulation mechanism,
while fine-grained textual details are concatenated with the music patch
sequence as inputs. Through an in-depth study, we demonstrate that rectified
flow training with an optimized architecture significantly outperforms
established diffusion methods for the text-to-music task, as evidenced by
various automatic metrics and human preference evaluations. Our experimental
data, code, and model weights are made publicly available at:
https://github.com/feizc/FluxMusic.
66d7fd90a5098dc770352d7d
Zhengcong Fei
admin_assigned
2024-09-04T08:38:48.396Z
617ba1820e4237bd1731b867
/avatars/f9de06363e64bddd7dc977e96e85df8a.svg
zhengcong fei
user
onion
66d7fd90a5098dc770352d7e
Mingyuan Fan
admin_assigned
2024-09-04T08:38:59.460Z
634672bfb7b4e71c7f45360f
/avatars/4b646fc3e271be90b9ec619d42ce3e99.svg
Fan Mingyuan
user
MichaelFan
66d7fd90a5098dc770352d7f
Changqian Yu
admin_assigned
2024-09-04T08:39:08.151Z
6419e0b3ed725fef6444f53a
https://cdn-avatars.huggingface.co/v1/production/uploads/6419e0b3ed725fef6444f53a/3ZzIafBJa50-gAu1PASuG.png
Yu Changqian
user
Changqian
66d7fd90a5098dc770352d80
Junshi Huang
admin_assigned
2024-09-04T08:39:18.529Z
642e8321be01b88c9447b25e
/avatars/cc760d9b1bd943da1e6d21f14cc6d495.svg
HuangJunshi
user
huangjunshi
FLUX that Plays Music
This paper explores a simple extension of diffusion-based rectified flow
Transformers for text-to-music generation, termed as FluxMusic. Generally,
along with design in advanced
Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
into a latent VAE space of mel-spectrum. It involves first applying a sequence
of independent attention to the double text-music stream, followed by a stacked
single music stream for denoised patch prediction. We employ multiple
pre-trained text encoders to sufficiently capture caption semantic information
as well as inference flexibility. In between, coarse textual information, in
conjunction with time step embeddings, is utilized in a modulation mechanism,
while fine-grained textual details are concatenated with the music patch
sequence as inputs. Through an in-depth study, we demonstrate that rectified
flow training with an optimized architecture significantly outperforms
established diffusion methods for the text-to-music task, as evidenced by
various automatic metrics and human preference evaluations. Our experimental
data, code, and model weights are made publicly available at:
https://github.com/feizc/FluxMusic.
FluxMusic extends diffusion-based rectified flow Transformers to generate music from text by transforming it into a latent VAE space of mel-spectra and using text encoders and modulations for enhanced semantic capture.
diffusion-based rectified flow Transformers Flux latent VAE space mel-spectrum attention denoised patch prediction text encoders modulation mechanism automatic metrics human preference evaluations