Zero-shot Cross-lingual Voice Transfer for TTS
In this paper, we introduce a zero-shot Voice Transfer (VT) module that can
be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to
transfer an individual's voice across languages. Our proposed VT module
comprises a speaker-encoder that processes reference speech, a bottleneck
layer, and residual adapters, connected to preexisting TTS layers. We compare
the performance of various configurations of these components and report Mean
Opinion Score (MOS) and Speaker Similarity across languages. Using a single
English reference speech per speaker, we achieve an average voice transfer
similarity score of 73% across nine target languages. Vocal characteristics
contribute significantly to the construction and perception of individual
identity. The loss of one's voice, due to physical or neurological conditions,
can lead to a profound sense of loss, impacting one's core identity. As a case
study, we demonstrate that our approach can not only transfer typical speech
but also restore the voices of individuals with dysarthria, even when only
atypical speech samples are available - a valuable utility for those who have
never had typical speech or banked their voice. Cross-lingual typical audio
samples, plus videos demonstrating voice restoration for dysarthric speakers
are available here
(google.github.io/tacotron/publications/zero_shot_voice_transfer).
66f2408f51dca07c005bf987
Fadi Biadsy
66f2408f51dca07c005bf988
Youzheng Chen
66f2408f51dca07c005bf989
Isaac Elias
66f2408f51dca07c005bf98a
Kyle Kastner
66f2408f51dca07c005bf98b
Gary Wang
66f2408f51dca07c005bf98c
Andrew Rosenberg
66f2408f51dca07c005bf98d
Bhuvana Ramabhadran
Zero-shot Cross-lingual Voice Transfer for TTS
In this paper, we introduce a zero-shot Voice Transfer (VT) module that can
be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to
transfer an individual's voice across languages. Our proposed VT module
comprises a speaker-encoder that processes reference speech, a bottleneck
layer, and residual adapters, connected to preexisting TTS layers. We compare
the performance of various configurations of these components and report Mean
Opinion Score (MOS) and Speaker Similarity across languages. Using a single
English reference speech per speaker, we achieve an average voice transfer
similarity score of 73% across nine target languages. Vocal characteristics
contribute significantly to the construction and perception of individual
identity. The loss of one's voice, due to physical or neurological conditions,
can lead to a profound sense of loss, impacting one's core identity. As a case
study, we demonstrate that our approach can not only transfer typical speech
but also restore the voices of individuals with dysarthria, even when only
atypical speech samples are available - a valuable utility for those who have
never had typical speech or banked their voice. Cross-lingual typical audio
samples, plus videos demonstrating voice restoration for dysarthric speakers
are available here
(google.github.io/tacotron/publications/zero_shot_voice_transfer).
The zero-shot Voice Transfer module integrates into a multi-lingual TTS system to transfer a speaker's voice across languages using a speaker-encoder, bottleneck layer, and residual adapters.
zero-shot Voice Transfer VT module Text-to-speech TTS speaker-encoder bottleneck layer residual adapters Mean Opinion Score MOS Speaker Similarity voice transfer cross-lingual dysarthria