Qwen2-Audio Technical Report
We introduce the latest progress of Qwen-Audio, a large-scale audio-language
model called Qwen2-Audio, which is capable of accepting various audio signal
inputs and performing audio analysis or direct textual responses with regard to
speech instructions. In contrast to complex hierarchical tags, we have
simplified the pre-training process by utilizing natural language prompts for
different data and tasks, and have further expanded the data volume. We have
boosted the instruction-following capability of Qwen2-Audio and implemented two
distinct audio interaction modes for voice chat and audio analysis. In the
voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
without text input. In the audio analysis mode, users could provide audio and
text instructions for analysis during the interaction. Note that we do not use
any system prompts to switch between voice chat and audio analysis modes.
Qwen2-Audio is capable of intelligently comprehending the content within audio
and following voice commands to respond appropriately. For instance, in an
audio segment that simultaneously contains sounds, multi-speaker conversations,
and a voice command, Qwen2-Audio can directly understand the command and
provide an interpretation and response to the audio. Additionally, DPO has
optimized the model's performance in terms of factuality and adherence to
desired behavior. According to the evaluation results from AIR-Bench,
Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
focused on audio-centric instruction-following capabilities. Qwen2-Audio is
open-sourced with the aim of fostering the advancement of the multi-modal
language community.
66973aa13db47c0771e24d99
Yunfei Chu
admin_assigned
2024-07-22T07:19:50.377Z
62c6a751a71b40cf26f359a8
/avatars/49abd2e71946035452c316d703baaac6.svg
Yunfei Chu
user
faychu
66973aa13db47c0771e24d9a
Jin Xu
66973aa13db47c0771e24d9b
Qian Yang
66973aa13db47c0771e24d9c
Haojie Wei
admin_assigned
2024-07-22T07:21:40.644Z
63bd0f800608fbc76b837e3f
/avatars/b3089c6162e4db7ffbdd2700053c0e85.svg
Haojie Wei
user
Dream-High
66973aa13db47c0771e24d9d
Xipin Wei
66973aa13db47c0771e24d9e
Zhifang Guo
admin_assigned
2024-07-22T07:21:21.580Z
661e577cbac5d981f883b743
/avatars/95e55e9707a6b55594c264081202d7f4.svg
GuoZhifang
user
ZhifangGuo
66973aa13db47c0771e24d9f
Yichong Leng
admin_assigned
2024-07-22T07:21:14.428Z
64a0347b528a9bbe59d6e08c
/avatars/6dd0bad84d711d1048a0a4169e621773.svg
Yichong Leng
user
ustcscallion
66973aa13db47c0771e24da0
Yuanjun Lv
claimed_verified
2024-07-18T09:06:42.599Z
659e513ea9bc1f60189ac148
https://cdn-avatars.huggingface.co/v1/production/uploads/659e513ea9bc1f60189ac148/DBDDqjGTQ0SvjWyuYu7py.jpeg
YuanjunLv
user
Bakerbunker
66973aa13db47c0771e24da1
Jinzheng He
admin_assigned
2024-07-22T07:21:06.819Z
6594f06ac04427eb38444bce
/avatars/b13fbf589b25eff038deb3fa12d95871.svg
Jinzheng He
user
jinzheng-he
66973aa13db47c0771e24da2
Junyang Lin
admin_assigned
2024-07-22T07:20:59.566Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
66973aa13db47c0771e24da3
Chang Zhou
admin_assigned
2024-07-22T07:20:51.996Z
622892774323cef93a956a4a
/avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
chang zhou
user
jiemizc
66973aa13db47c0771e24da4
Jingren Zhou
admin_assigned
2024-07-22T07:20:36.040Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
Qwen2-Audio Technical Report
We introduce the latest progress of Qwen-Audio, a large-scale audio-language
model called Qwen2-Audio, which is capable of accepting various audio signal
inputs and performing audio analysis or direct textual responses with regard to
speech instructions. In contrast to complex hierarchical tags, we have
simplified the pre-training process by utilizing natural language prompts for
different data and tasks, and have further expanded the data volume. We have
boosted the instruction-following capability of Qwen2-Audio and implemented two
distinct audio interaction modes for voice chat and audio analysis. In the
voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
without text input. In the audio analysis mode, users could provide audio and
text instructions for analysis during the interaction. Note that we do not use
any system prompts to switch between voice chat and audio analysis modes.
Qwen2-Audio is capable of intelligently comprehending the content within audio
and following voice commands to respond appropriately. For instance, in an
audio segment that simultaneously contains sounds, multi-speaker conversations,
and a voice command, Qwen2-Audio can directly understand the command and
provide an interpretation and response to the audio. Additionally, DPO has
optimized the model's performance in terms of factuality and adherence to
desired behavior. According to the evaluation results from AIR-Bench,
Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
focused on audio-centric instruction-following capabilities. Qwen2-Audio is
open-sourced with the aim of fostering the advancement of the multi-modal
language community.
Qwen2-Audio, a large-scale audio-language model, enhances instruction-following and audio analysis through natural language prompts and DPO optimization.
audio-language model pre-training process natural language prompts instruction-following capability voice chat mode audio analysis mode multi-speaker conversations voice commands DPO audio-centric instruction-following AIR-Bench multi-modal language community