Towards General-Purpose Speech Abilities for Large Language Models Using
  Unpaired Data
In this work, we extend the instruction-tuned Llama-2 model with end-to-end
general-purpose speech processing and reasoning abilities while maintaining the
wide range of LLM capabilities, without using any carefully curated paired
data. The proposed model can utilize audio prompts as a replacement for text
and sustain a conversation. Such a model also has extended cross-modal
capabilities such as being able to perform speech question answering, speech
translation, and audio summarization amongst many other closed and open-domain
tasks. This is unlike prior approaches in speech, in which LLMs are extended to
handle audio for a limited number of pre-designated tasks. Experiments show
that our end-to-end approach is on par with or outperforms a cascaded system
(speech recognizer + LLM) in terms of modeling the response to a prompt.
Furthermore, unlike a cascade, our approach shows the ability to interchange
text and audio modalities and utilize the prior context in a conversation to
provide better results.
6552edb8f2a8f0b4c6cc0c8b
Yassir Fathullah
admin_assigned
2023-11-14T10:52:51.763Z
64106057b27543634e3922bd
/avatars/c4eea7d419453802c67c4f7355315cac.svg
Yassir Fathullah
user
yassirf
6552edb8f2a8f0b4c6cc0c8c
Chunyang Wu
admin_assigned
2023-11-14T10:53:03.412Z
641d8fb516ef0e5a7274cf1a
/avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
wuchunyang
user
Picc
6552edb8f2a8f0b4c6cc0c8d
Egor Lakomkin
admin_assigned
2023-11-14T10:53:10.687Z
61e1bb4a210b4f412674d08d
/avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
Egor Lakomkin
user
EgorL
6552edb8f2a8f0b4c6cc0c8e
Junteng Jia
admin_assigned
2023-11-14T10:53:23.148Z
6171c03ff37773b575f8b39f
/avatars/feb21515dfd111fe6a222469ea7d6950.svg
Junteng Jia
user
JuntengJia
6552edb8f2a8f0b4c6cc0c8f
Yuan Shangguan
6552edb8f2a8f0b4c6cc0c90
Jay Mahadeokar
6552edb8f2a8f0b4c6cc0c91
Ozlem Kalinli
6552edb8f2a8f0b4c6cc0c92
Christian Fuegen
6552edb8f2a8f0b4c6cc0c93
Mike Seltzer
Towards General-Purpose Speech Abilities for Large Language Models Using
  Unpaired Data
In this work, we extend the instruction-tuned Llama-2 model with end-to-end
general-purpose speech processing and reasoning abilities while maintaining the
wide range of LLM capabilities, without using any carefully curated paired
data. The proposed model can utilize audio prompts as a replacement for text
and sustain a conversation. Such a model also has extended cross-modal
capabilities such as being able to perform speech question answering, speech
translation, and audio summarization amongst many other closed and open-domain
tasks. This is unlike prior approaches in speech, in which LLMs are extended to
handle audio for a limited number of pre-designated tasks. Experiments show
that our end-to-end approach is on par with or outperforms a cascaded system
(speech recognizer + LLM) in terms of modeling the response to a prompt.
Furthermore, unlike a cascade, our approach shows the ability to interchange
text and audio modalities and utilize the prior context in a conversation to
provide better results.
An end-to-end extended Llama-2 model incorporating speech processing and reasoning supports a variety of tasks with audio and text, matching or outperforming cascaded systems in response modeling and context utilization.
instruction-tuned Llama-2 end-to-end speech processing reasoning abilities LLM capabilities audio prompts conversation speech question answering speech translation audio summarization cross-modal capabilities cascaded system speech recognizer text and audio modalities prior context