AudioPaLM: A Large Language Model That Can Speak and Listen
We introduce AudioPaLM, a large language model for speech understanding and
generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
[Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
multimodal architecture that can process and generate text and speech with
applications including speech recognition and speech-to-speech translation.
AudioPaLM inherits the capability to preserve paralinguistic information such
as speaker identity and intonation from AudioLM and the linguistic knowledge
present only in text large language models such as PaLM-2. We demonstrate that
initializing AudioPaLM with the weights of a text-only large language model
improves speech processing, successfully leveraging the larger quantity of text
training data used in pretraining to assist with the speech tasks. The
resulting model significantly outperforms existing systems for speech
translation tasks and has the ability to perform zero-shot speech-to-text
translation for many languages for which input/target language combinations
were not seen in training. AudioPaLM also demonstrates features of audio
language models, such as transferring a voice across languages based on a short
spoken prompt. We release examples of our method at
https://google-research.github.io/seanet/audiopalm/examples
6494ef61771d68df8d060909
Paul K. Rubenstein
6494ef61771d68df8d06090a
Chulayuth Asawaroengchai
6494ef61771d68df8d06090b
Duc Dung Nguyen
6494ef61771d68df8d06090c
Ankur Bapna
admin_assigned
2023-06-23T12:09:54.990Z
62cdba279403ec4719d384f8
/avatars/c62c7d16b5cf382cdeb7fbfb5e291917.svg
Ankur Bapna
user
ankurbpn
6494ef61771d68df8d06090d
Zalán Borsos
6494ef61771d68df8d06090e
Félix de Chaumont Quitry
6494ef61771d68df8d06090f
Peter Chen
6494ef61771d68df8d060910
Dalia El Badawy
6494ef61771d68df8d060911
Wei Han
claimed_verified
2023-06-23T07:31:27.769Z
6489e3992f9e11a0c26973b6
/avatars/e637e6caab6df56da6005f73c613a3d0.svg
Wei Han
user
weihan3
6494ef61771d68df8d060912
Eugene Kharitonov
admin_assigned
2023-06-23T12:11:20.840Z
64509292fc5e5ff919b4a97d
https://cdn-avatars.huggingface.co/v1/production/uploads/64509292fc5e5ff919b4a97d/OOn5EaCXw39ePmgKptOuT.png
Eugene Kharitonov
user
ekharitonov
6494ef61771d68df8d060913
Hannah Muckenhirn
6494ef61771d68df8d060914
Dirk Padfield
6494ef61771d68df8d060915
James Qin
6494ef61771d68df8d060916
Danny Rozenberg
6494ef61771d68df8d060917
Tara Sainath
6494ef61771d68df8d060918
Johan Schalkwyk
6494ef61771d68df8d060919
Matt Sharifi
6494ef61771d68df8d06091a
Michelle Tadmor Ramanovich
6494ef61771d68df8d06091c
Marco Tagliasacchi
6494ef61771d68df8d06091d
Alexandru Tudor
6494ef61771d68df8d06091e
Mihajlo Velimirović
6494ef61771d68df8d06091f
Damien Vincent
6494ef61771d68df8d060920
Jiahui Yu
admin_assigned
2023-06-23T12:18:09.782Z
6174394801833e378dfc2517
/avatars/4989bcdda7ba1ae51cab8235c3e39275.svg
Jiahui Yu
user
jiahuiyu
6494ef61771d68df8d060921
Yongqiang Wang
6494ef61771d68df8d060922
Vicky Zayats
6494ef61771d68df8d060923
Neil Zeghidour
6494ef61771d68df8d060924
Yu Zhang
6494ef61771d68df8d060925
Zhishuai Zhang
6494ef61771d68df8d060926
Lukas Zilka
6494ef61771d68df8d060927
Christian Frank
AudioPaLM: A Large Language Model That Can Speak and Listen
We introduce AudioPaLM, a large language model for speech understanding and
generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
[Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
multimodal architecture that can process and generate text and speech with
applications including speech recognition and speech-to-speech translation.
AudioPaLM inherits the capability to preserve paralinguistic information such
as speaker identity and intonation from AudioLM and the linguistic knowledge
present only in text large language models such as PaLM-2. We demonstrate that
initializing AudioPaLM with the weights of a text-only large language model
improves speech processing, successfully leveraging the larger quantity of text
training data used in pretraining to assist with the speech tasks. The
resulting model significantly outperforms existing systems for speech
translation tasks and has the ability to perform zero-shot speech-to-text
translation for many languages for which input/target language combinations
were not seen in training. AudioPaLM also demonstrates features of audio
language models, such as transferring a voice across languages based on a short
spoken prompt. We release examples of our method at
https://google-research.github.io/seanet/audiopalm/examples
A unified multimodal language model combining text and speech capabilities outperforms existing systems in speech translation and zero-shot speech-to-text translation.
AudioPaLM PaLM-2 AudioLM multimodal architecture paralinguistic information speaker identity intonation speech recognition speech-to-speech translation zero-shot speech-to-text translation audio language models voice transfer