CLAPSpeech: Learning Prosody from Text Context with Contrastive
  Language-Audio Pre-training
Improving text representation has attracted much attention to achieve
expressive text-to-speech (TTS). However, existing works only implicitly learn
the prosody with masked token reconstruction tasks, which leads to low training
efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
cross-modal contrastive pre-training framework that explicitly learns the
prosody variance of the same text token under different contexts. Specifically,
1) We encourage the model to connect the text context with its corresponding
prosody pattern in the joint multi-modal space with the elaborate design of the
encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
pipeline to capture prosody patterns in multiple levels. We show how to
incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
on three datasets not only show that CLAPSpeech could improve the prosody
prediction for existing TTS methods, but also demonstrate its generalization
ability to adapt to multiple languages and multi-speaker TTS. We also deeply
analyze the principle behind the performance of CLAPSpeech. Ablation studies
demonstrate the necessity of each component in our method. Source code and
audio samples are available at https://clapspeech.github.io.
646700eff3e78d1d6be121a3
Zhenhui Ye
admin_assigned
2023-05-21T20:41:10.207Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
646700eff3e78d1d6be121a4
Rongjie Huang
admin_assigned
2023-05-21T20:41:54.058Z
62f6556e04e5e02f82ac8531
/avatars/01024815aeaaf8c4721e937b3db68903.svg
Rongjiehuang
user
Rongjiehuang
646700eff3e78d1d6be121a5
Yi Ren
646700eff3e78d1d6be121a6
Ziyue Jiang
646700eff3e78d1d6be121a7
Jinglin Liu
admin_assigned
2023-05-21T20:42:36.814Z
6205c6ad00e630c7a5d0fc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
Jinglin Liu
user
Silentlin
646700eff3e78d1d6be121a8
Jinzheng He
646700eff3e78d1d6be121a9
Xiang Yin
646700eff3e78d1d6be121aa
Zhou Zhao
CLAPSpeech: Learning Prosody from Text Context with Contrastive
  Language-Audio Pre-training
Improving text representation has attracted much attention to achieve
expressive text-to-speech (TTS). However, existing works only implicitly learn
the prosody with masked token reconstruction tasks, which leads to low training
efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
cross-modal contrastive pre-training framework that explicitly learns the
prosody variance of the same text token under different contexts. Specifically,
1) We encourage the model to connect the text context with its corresponding
prosody pattern in the joint multi-modal space with the elaborate design of the
encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
pipeline to capture prosody patterns in multiple levels. We show how to
incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
on three datasets not only show that CLAPSpeech could improve the prosody
prediction for existing TTS methods, but also demonstrate its generalization
ability to adapt to multiple languages and multi-speaker TTS. We also deeply
analyze the principle behind the performance of CLAPSpeech. Ablation studies
demonstrate the necessity of each component in our method. Source code and
audio samples are available at https://clapspeech.github.io.
CLAPSpeech is a cross-modal contrastive pre-training framework that enhances prosody in text-to-speech systems by learning prosodic variance across contexts.
cross-modal contrastive pre-training prosody variance encoder inputs contrastive loss multi-scale pre-training prosody prediction multi-speaker TTS