UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Language models (LMs) have demonstrated the capability to handle a variety of
generative tasks. This paper presents the UniAudio system, which, unlike prior
task-specific approaches, leverages LMs techniques to generate multiple types
of audio (including speech, sounds, music, and singing) with given input
conditions. UniAudio 1) first tokenizes all types of target audio along with
other condition modalities, 2) concatenates source-target pair as a single
sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale
Transformer model is proposed to handle the overly long sequences caused by the
residual vector quantization based neural codec in tokenization. Training of
UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all
generative tasks, aiming to obtain sufficient prior knowledge not only in the
intrinsic properties of audio but also the inter-relationship between audio and
other modalities. Therefore, the trained UniAudio model has the potential to
become a foundation model for universal audio generation: it shows strong
capability in all trained tasks and can seamlessly support new audio generation
tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves
state-of-the-art or at least competitive results on most of the 11 tasks. Demo
and code are released at https://github.com/yangdongchao/UniAudio
651ba882e315e8a224a0a530
Dongchao Yang
claimed_verified
2023-10-04T07:37:38.046Z
63c7636b656e7822e23e6f6b
/avatars/41bfd5e1ce6daab6058eacfd33c7a268.svg
Dongchao Yang
user
Dongchao
651ba882e315e8a224a0a531
Jinchuan Tian
admin_assigned
2023-10-09T09:02:25.455Z
64770ff6d7cf39f2e937b882
/avatars/7b729501856f5073cf9e484811012f2a.svg
Jinchuan Tian
user
JinchuanTian
651ba882e315e8a224a0a532
Xu Tan
admin_assigned
2023-10-09T09:02:40.924Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
651ba882e315e8a224a0a533
Rongjie Huang
admin_assigned
2023-10-09T09:03:01.114Z
62f6556e04e5e02f82ac8531
/avatars/01024815aeaaf8c4721e937b3db68903.svg
Rongjiehuang
user
Rongjiehuang
651ba882e315e8a224a0a534
Songxiang Liu
admin_assigned
2023-10-09T09:03:13.379Z
626b4350822e3d85324c6054
/avatars/3e27115ea73f33ba40e8599cdf2725af.svg
liusongxiang
user
liusongxiang
651ba882e315e8a224a0a535
Xuankai Chang
admin_assigned
2023-10-09T09:03:21.023Z
6136b5707c6d9b8e41469bec
/avatars/5b7f4ee0ba36bb6b87accda0ee27bb17.svg
Xuankai Chang
user
simpleoier
651ba882e315e8a224a0a536
Jiatong Shi
admin_assigned
2023-10-09T09:03:29.339Z
6136ec7a40e43b8f748a0832
/avatars/089afa82ecfe6432d2dbfe81048c6ffa.svg
Jiatong Shi
user
ftshijt
651ba882e315e8a224a0a537
Sheng Zhao
651ba882e315e8a224a0a538
Jiang Bian
admin_assigned
2023-10-09T09:04:12.094Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
651ba882e315e8a224a0a539
Xixin Wu
651ba882e315e8a224a0a53a
Zhou Zhao
651ba882e315e8a224a0a53b
Helen Meng
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Language models (LMs) have demonstrated the capability to handle a variety of
generative tasks. This paper presents the UniAudio system, which, unlike prior
task-specific approaches, leverages LMs techniques to generate multiple types
of audio (including speech, sounds, music, and singing) with given input
conditions. UniAudio 1) first tokenizes all types of target audio along with
other condition modalities, 2) concatenates source-target pair as a single
sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale
Transformer model is proposed to handle the overly long sequences caused by the
residual vector quantization based neural codec in tokenization. Training of
UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all
generative tasks, aiming to obtain sufficient prior knowledge not only in the
intrinsic properties of audio but also the inter-relationship between audio and
other modalities. Therefore, the trained UniAudio model has the potential to
become a foundation model for universal audio generation: it shows strong
capability in all trained tasks and can seamlessly support new audio generation
tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves
state-of-the-art or at least competitive results on most of the 11 tasks. Demo
and code are released at https://github.com/yangdongchao/UniAudio
UniAudio, a unified language model system, generates various audio types effectively using tokenization, sequence concatenation, and next-token prediction, with a multi-scale Transformer model handling extended sequences.
language models tokenization next-token prediction multi-scale Transformer model residual vector quantization neural codec universal audio generation fine-tuning