Audiobox: Unified Audio Generation with Natural Language Prompts
Audio is an essential part of our life, but creating it often requires
expertise and is time-consuming. Research communities have made great progress
over the past year advancing the performance of large scale audio generative
models for a single modality (speech, sound, or music) through adopting more
powerful generative models and scaling data. However, these models lack
controllability in several aspects: speech generation models cannot synthesize
novel styles based on text description and are limited on domain coverage such
as outdoor environments; sound generation models only provide coarse-grained
control based on descriptions like "a person speaking" and would only generate
mumbling human voices. This paper presents Audiobox, a unified model based on
flow-matching that is capable of generating various audio modalities. We design
description-based and example-based prompting to enhance controllability and
unify speech and sound generation paradigms. We allow transcript, vocal, and
other audio styles to be controlled independently when generating speech. To
improve model generalization with limited labels, we adapt a self-supervised
infilling objective to pre-train on large quantities of unlabeled audio.
Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
unlocks new methods for generating audio with novel vocal and acoustic styles.
We further integrate Bespoke Solvers, which speeds up generation by over 25
times compared to the default ODE solver for flow-matching, without loss of
performance on several tasks. Our demo is available at
https://audiobox.metademolab.com/
658bb8df1783d1ebc7639690
Apoorv Vyas
658bb8df1783d1ebc7639691
Bowen Shi
658bb8df1783d1ebc7639692
Matthew Le
658bb8df1783d1ebc7639693
Andros Tjandra
claimed_verified
2025-02-13T20:36:52.153Z
647779c6c21496284db45835
/avatars/c1176e57f5d0aaa5f50bbde48a19af01.svg
Andros Tjandra
user
androstj
658bb8df1783d1ebc7639694
Yi-Chiao Wu
658bb8df1783d1ebc7639695
Baishan Guo
658bb8df1783d1ebc7639696
Jiemin Zhang
658bb8df1783d1ebc7639697
Xinyue Zhang
658bb8df1783d1ebc7639698
Robert Adkins
658bb8df1783d1ebc7639699
William Ngan
658bb8df1783d1ebc763969a
Jeff Wang
658bb8df1783d1ebc763969b
Ivan Cruz
658bb8df1783d1ebc763969c
Bapi Akula
658bb8df1783d1ebc763969d
Akinniyi Akinyemi
658bb8df1783d1ebc763969e
Brian Ellis
658bb8df1783d1ebc763969f
Rashel Moritz
658bb8df1783d1ebc76396a0
Yael Yungster
658bb8df1783d1ebc76396a1
Alice Rakotoarison
658bb8df1783d1ebc76396a2
Liang Tan
658bb8df1783d1ebc76396a3
Chris Summers
658bb8df1783d1ebc76396a4
Carleigh Wood
658bb8df1783d1ebc76396a5
Joshua Lane
658bb8df1783d1ebc76396a6
Mary Williamson
658bb8df1783d1ebc76396a7
Wei-Ning Hsu
Audiobox: Unified Audio Generation with Natural Language Prompts
Audio is an essential part of our life, but creating it often requires
expertise and is time-consuming. Research communities have made great progress
over the past year advancing the performance of large scale audio generative
models for a single modality (speech, sound, or music) through adopting more
powerful generative models and scaling data. However, these models lack
controllability in several aspects: speech generation models cannot synthesize
novel styles based on text description and are limited on domain coverage such
as outdoor environments; sound generation models only provide coarse-grained
control based on descriptions like "a person speaking" and would only generate
mumbling human voices. This paper presents Audiobox, a unified model based on
flow-matching that is capable of generating various audio modalities. We design
description-based and example-based prompting to enhance controllability and
unify speech and sound generation paradigms. We allow transcript, vocal, and
other audio styles to be controlled independently when generating speech. To
improve model generalization with limited labels, we adapt a self-supervised
infilling objective to pre-train on large quantities of unlabeled audio.
Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
unlocks new methods for generating audio with novel vocal and acoustic styles.
We further integrate Bespoke Solvers, which speeds up generation by over 25
times compared to the default ODE solver for flow-matching, without loss of
performance on several tasks. Our demo is available at
https://audiobox.metademolab.com/
Audiobox, a flow-matching based unified model, enhances controllability and efficiency in generating various audio modalities including speech and sound using description-based and example-based prompting.
flow-matching description-based prompting example-based prompting speech generation sound generation self-supervised infilling zero-shot TTS text-to-sound Librispeech AudioCaps Bespoke Solvers ODE solver