SALMONN: Towards Generic Hearing Abilities for Large Language Models
Hearing is arguably an essential ability of artificial intelligence (AI)
agents in the physical world, which refers to the perception and understanding
of general auditory information consisting of at least three types of sounds:
speech, audio events, and music. In this paper, we propose SALMONN, a speech
audio language music open neural network, built by integrating a pre-trained
text-based large language model (LLM) with speech and audio encoders into a
single multimodal model. SALMONN enables the LLM to directly process and
understand general audio inputs and achieve competitive performances on a
number of speech and audio tasks used in training, such as automatic speech
recognition and translation, auditory-information-based question answering,
emotion recognition, speaker verification, and music and audio captioning
etc. SALMONN also has a diverse set of emergent abilities unseen in
the training, which includes but is not limited to speech translation to
untrained languages, speech-based slot filling, spoken-query-based question
answering, audio-based storytelling, and speech audio co-reasoning
etc. The presence of the cross-modal emergent abilities is studied,
and a novel few-shot activation tuning approach is proposed to activate such
abilities of SALMONN. To our knowledge, SALMONN is the first model of its type
and can be regarded as a step towards AI with generic hearing abilities. An
interactive demo of SALMONN is available at
\url{https://github.com/bytedance/SALMONN}, and the training code and
model checkpoints will be released upon acceptance.
6535eb4699d8bba294550d3c
Changli Tang
admin_assigned
2023-10-23T08:55:43.692Z
63770389cdcc1bf630870758
/avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
Changli Tang
user
Changli
6535eb4699d8bba294550d3d
Wenyi Yu
6535eb4699d8bba294550d3e
Guangzhi Sun
admin_assigned
2023-10-23T08:56:14.406Z
64ab544489aa67e4a2505eeb
/avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
Guangzhi Sun
user
BrianatCambridge
6535eb4699d8bba294550d3f
Xianzhao Chen
admin_assigned
2023-10-23T08:56:56.476Z
646617cfcf550af36eb2e773
/avatars/963271ee1a69b5524dafe5d90c93ff3f.svg
chenxianzhao
user
chenxianzhao
6535eb4699d8bba294550d40
Tian Tan
6535eb4699d8bba294550d41
Wei Li
6535eb4699d8bba294550d42
Lu Lu
6535eb4699d8bba294550d43
Zejun Ma
admin_assigned
2023-10-23T08:58:02.291Z
64cfc1d4228324a28bd411c2
/avatars/2d53afb1f32e8620568862681ce3bd27.svg
MA
user
Zejun68
6535eb4699d8bba294550d44
Chao Zhang
claimed_verified
2023-10-23T09:43:47.915Z
65363f359326d6da5f651f2e
/avatars/13c0dce7c5ee2f1bd5eb451e8707979e.svg
Chao Zhang
user
cz277
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Hearing is arguably an essential ability of artificial intelligence (AI)
agents in the physical world, which refers to the perception and understanding
of general auditory information consisting of at least three types of sounds:
speech, audio events, and music. In this paper, we propose SALMONN, a speech
audio language music open neural network, built by integrating a pre-trained
text-based large language model (LLM) with speech and audio encoders into a
single multimodal model. SALMONN enables the LLM to directly process and
understand general audio inputs and achieve competitive performances on a
number of speech and audio tasks used in training, such as automatic speech
recognition and translation, auditory-information-based question answering,
emotion recognition, speaker verification, and music and audio captioning
etc. SALMONN also has a diverse set of emergent abilities unseen in
the training, which includes but is not limited to speech translation to
untrained languages, speech-based slot filling, spoken-query-based question
answering, audio-based storytelling, and speech audio co-reasoning
etc. The presence of the cross-modal emergent abilities is studied,
and a novel few-shot activation tuning approach is proposed to activate such
abilities of SALMONN. To our knowledge, SALMONN is the first model of its type
and can be regarded as a step towards AI with generic hearing abilities. An
interactive demo of SALMONN is available at
\url{https://github.com/bytedance/SALMONN}, and the training code and
model checkpoints will be released upon acceptance.
SALMONN, an integrated multimodal model combining a pre-trained text-based LLM with speech and audio encoders, demonstrates competitive performance and emergent abilities in various speech and audio tasks.
pre-trained text-based large language model (LLM) speech encoders audio encoders multimodal model automatic speech recognition translation auditory-information-based question answering emotion recognition speaker verification music captioning audio captioning speech translation speech-based slot filling spoken-query-based question answering audio-based storytelling speech audio co-reasoning few-shot activation tuning