Qwen-Audio: Advancing Universal Audio Understanding via Unified
  Large-Scale Audio-Language Models
Recently, instruction-following audio-language models have received broad
attention for audio interaction with humans. However, the absence of
pre-trained audio models capable of handling diverse audio types and tasks has
hindered progress in this field. Consequently, most existing works have only
been able to support a limited range of interaction capabilities. In this
paper, we develop the Qwen-Audio model and address this limitation by scaling
up audio-language pre-training to cover over 30 tasks and various audio types,
such as human speech, natural sounds, music, and songs, to facilitate universal
audio understanding abilities. However, directly co-training all tasks and
datasets can lead to interference issues, as the textual labels associated with
different datasets exhibit considerable variations due to differences in task
focus, language, granularity of annotation, and text structure. To overcome the
one-to-many interference, we carefully design a multi-task training framework
by conditioning on a sequence of hierarchical tags to the decoder for
encouraging knowledge sharing and avoiding interference through shared and
specified tags respectively. Remarkably, Qwen-Audio achieves impressive
performance across diverse benchmark tasks without requiring any task-specific
fine-tuning, surpassing its counterparts. Building upon the capabilities of
Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from
various audios and text inputs, enabling multi-turn dialogues and supporting
various audio-central scenarios.
655442a6e149dd8e9f30702a
Yunfei Chu
admin_assigned
2023-11-15T10:15:38.617Z
62c6a751a71b40cf26f359a8
/avatars/49abd2e71946035452c316d703baaac6.svg
Yunfei Chu
user
faychu
655442a6e149dd8e9f30702b
Jin Xu
655442a6e149dd8e9f30702c
Xiaohuan Zhou
claimed_verified
2025-03-04T08:54:18.487Z
668f5875b5b3081d776e4094
/avatars/8c763393f25afbe5fb8b132f775e746a.svg
Xiaohuan Zhou
user
XiaohuanZhou
655442a6e149dd8e9f30702d
Qian Yang
655442a6e149dd8e9f30702e
Shiliang Zhang
655442a6e149dd8e9f30702f
Zhijie Yan
655442a6e149dd8e9f307030
Chang Zhou
admin_assigned
2023-11-15T10:12:53.677Z
622892774323cef93a956a4a
/avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
chang zhou
user
jiemizc
655442a6e149dd8e9f307031
Jingren Zhou
admin_assigned
2023-11-15T10:12:35.277Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
Qwen-Audio: Advancing Universal Audio Understanding via Unified
  Large-Scale Audio-Language Models
Recently, instruction-following audio-language models have received broad
attention for audio interaction with humans. However, the absence of
pre-trained audio models capable of handling diverse audio types and tasks has
hindered progress in this field. Consequently, most existing works have only
been able to support a limited range of interaction capabilities. In this
paper, we develop the Qwen-Audio model and address this limitation by scaling
up audio-language pre-training to cover over 30 tasks and various audio types,
such as human speech, natural sounds, music, and songs, to facilitate universal
audio understanding abilities. However, directly co-training all tasks and
datasets can lead to interference issues, as the textual labels associated with
different datasets exhibit considerable variations due to differences in task
focus, language, granularity of annotation, and text structure. To overcome the
one-to-many interference, we carefully design a multi-task training framework
by conditioning on a sequence of hierarchical tags to the decoder for
encouraging knowledge sharing and avoiding interference through shared and
specified tags respectively. Remarkably, Qwen-Audio achieves impressive
performance across diverse benchmark tasks without requiring any task-specific
fine-tuning, surpassing its counterparts. Building upon the capabilities of
Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from
various audios and text inputs, enabling multi-turn dialogues and supporting
various audio-central scenarios.
Qwen-Audio addresses the limitation of diverse audio types and tasks in instruction-following models through a multi-task training framework with hierarchical tagging, achieving superior performance across benchmarks and enabling Qwen-Audio-Chat for versatile dialogue scenarios.
instruction-following audio-language models pre-trained audio models multi-task training framework hierarchical tags knowledge sharing Qwen-Audio Qwen-Audio-Chat multi-turn dialogues