High-Fidelity Audio Compression with Improved RVQGAN
Language models have been successfully used to model natural signals, such as
images, speech, and music. A key component of these models is a high quality
neural compression model that can compress high-dimensional natural signals
into lower dimensional discrete tokens. To that end, we introduce a
high-fidelity universal neural audio compression algorithm that achieves ~90x
compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve
this by combining advances in high-fidelity audio generation with better vector
quantization techniques from the image domain, along with improved adversarial
and reconstruction losses. We compress all domains (speech, environment, music,
etc.) with a single universal model, making it widely applicable to generative
modeling of all audio. We compare with competing audio compression algorithms,
and find our method outperforms them significantly. We provide thorough
ablations for every design choice, as well as open-source code and trained
model weights. We hope our work can lay the foundation for the next generation
of high-fidelity audio modeling.
6487c6a66d93bc7eb47d1e97
Rithesh Kumar
6487c6a66d93bc7eb47d1e98
Prem Seetharaman
admin_assigned
2023-06-14T12:20:02.445Z
6009e97993286684ae332cdd
/avatars/2653d0809cba9f01a69a314e5a946f45.svg
Prem Seetharaman
user
pseeth
6487c6a66d93bc7eb47d1e99
Alejandro Luebs
admin_assigned
2023-06-14T12:20:19.592Z
63f667554b831cc179c01c28
/avatars/4c8029acacb96c5d70638a72ebdd5c01.svg
Alejandro Luebs
user
aluebs
6487c6a66d93bc7eb47d1e9a
Ishaan Kumar
claimed_verified
2023-06-14T14:54:16.220Z
6303d3e17b50dd9d0a35b7ff
/avatars/e9462ede328a7cac05d7d08f0a9961d4.svg
IshaanKumar
user
eeishaan
6487c6a66d93bc7eb47d1e9b
Kundan Kumar
High-Fidelity Audio Compression with Improved RVQGAN
Language models have been successfully used to model natural signals, such as
images, speech, and music. A key component of these models is a high quality
neural compression model that can compress high-dimensional natural signals
into lower dimensional discrete tokens. To that end, we introduce a
high-fidelity universal neural audio compression algorithm that achieves ~90x
compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve
this by combining advances in high-fidelity audio generation with better vector
quantization techniques from the image domain, along with improved adversarial
and reconstruction losses. We compress all domains (speech, environment, music,
etc.) with a single universal model, making it widely applicable to generative
modeling of all audio. We compare with competing audio compression algorithms,
and find our method outperforms them significantly. We provide thorough
ablations for every design choice, as well as open-source code and trained
model weights. We hope our work can lay the foundation for the next generation
of high-fidelity audio modeling.
A universal neural audio compression algorithm achieves high fidelity at 8 kbps bandwidth by combining advancements in audio generation, vector quantization, and improved loss functions.
neural compression model high-fidelity audio generation vector quantization adversarial loss reconstruction loss universal model