PolyVoice: Language Models for Speech to Speech Translation
We propose PolyVoice, a language model-based framework for speech-to-speech
translation (S2ST) system. Our framework consists of two language models: a
translation language model and a speech synthesis language model. We use
discretized speech units, which are generated in a fully unsupervised way, and
thus our framework can be used for unwritten languages. For the speech
synthesis part, we adopt the existing VALL-E X approach and build a unit-based
audio language model. This grants our framework the ability to preserve the
voice characteristics and the speaking style of the original speech. We examine
our system on Chinese rightarrow English and English rightarrow Spanish
pairs. Experimental results show that our system can generate speech with high
translation quality and audio quality. Speech samples are available at
https://speechtranslation.github.io/polyvoice.
647eab8f18274bce03088d01
Qianqian Dong
claimed_verified
2023-06-07T10:09:51.551Z
63e31b765dd0688d9363a398
/avatars/60d92bef43a7d1342f12ef5ac984f025.svg
Qianqian Dong
user
QQD
647eab8f18274bce03088d02
Zhiying Huang
647eab8f18274bce03088d03
Chen Xu
647eab8f18274bce03088d04
Yunlong Zhao
647eab8f18274bce03088d05
Kexin Wang
647eab8f18274bce03088d06
Xuxin Cheng
647eab8f18274bce03088d07
Tom Ko
647eab8f18274bce03088d08
Qiao Tian
647eab8f18274bce03088d09
Tang Li
647eab8f18274bce03088d0a
Fengpeng Yue
647eab8f18274bce03088d0b
Ye Bai
647eab8f18274bce03088d0c
Xi Chen
647eab8f18274bce03088d0d
Lu Lu
647eab8f18274bce03088d0e
Zejun Ma
647eab8f18274bce03088d0f
Yuping Wang
647eab8f18274bce03088d10
Mingxuan Wang
647eab8f18274bce03088d11
Yuxuan Wang
PolyVoice: Language Models for Speech to Speech Translation
We propose PolyVoice, a language model-based framework for speech-to-speech
translation (S2ST) system. Our framework consists of two language models: a
translation language model and a speech synthesis language model. We use
discretized speech units, which are generated in a fully unsupervised way, and
thus our framework can be used for unwritten languages. For the speech
synthesis part, we adopt the existing VALL-E X approach and build a unit-based
audio language model. This grants our framework the ability to preserve the
voice characteristics and the speaking style of the original speech. We examine
our system on Chinese rightarrow English and English rightarrow Spanish
pairs. Experimental results show that our system can generate speech with high
translation quality and audio quality. Speech samples are available at
https://speechtranslation.github.io/polyvoice.
PolyVoice, a language model-based framework for speech-to-speech translation, uses discretized speech units and VALL-E X for high-quality voice and style preservation in translation.
language model-based framework speech-to-speech translation translation language model speech synthesis language model discretized speech units unsupervised generation VALL-E X unit-based audio language model