HierSpeech++: Bridging the Gap between Semantic and Acoustic
  Representation of Speech by Hierarchical Variational Inference for Zero-shot
  Speech Synthesis
Large language models (LLM)-based speech synthesis has been widely adopted in
zero-shot speech synthesis. However, they require a large-scale data and
possess the same limitations as previous autoregressive speech models,
including slow inference speed and lack of robustness. This paper proposes
HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech
(TTS) and voice conversion (VC). We verified that hierarchical speech synthesis
frameworks could significantly improve the robustness and expressiveness of the
synthetic speech. Furthermore, we significantly improve the naturalness and
speaker similarity of synthetic speech even in zero-shot speech synthesis
scenarios. For text-to-speech, we adopt the text-to-vec framework, which
generates a self-supervised speech representation and an F0 representation
based on text representations and prosody prompts. Then, HierSpeech++ generates
speech from the generated vector, F0, and voice prompt. We further introduce a
high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The
experimental results demonstrated that the hierarchical variational autoencoder
could be a strong zero-shot speech synthesizer given that it outperforms
LLM-based and diffusion-based models. Moreover, we achieved the first
human-level quality zero-shot speech synthesis. Audio samples and source code
are available at https://github.com/sh-lee-prml/HierSpeechpp.
655d90c9bf82497038e70802
Sang-Hoon Lee
admin_assigned
2023-11-22T12:50:49.532Z
654a62c211a85feec88ef8f2
/avatars/aeff2661e7f3cbc6b3cdf43c6e31ca9b.svg
Sang-Hoon Lee
user
LeeSangHoon
655d90c9bf82497038e70803
Ha-Yeong Choi
admin_assigned
2023-11-22T12:50:56.373Z
6401a0c14a5c92eccfe333c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6401a0c14a5c92eccfe333c2/-fv3jekUxMNrE058phGpA.png
Ha-Yeong Choi
user
Ha0
655d90c9bf82497038e70804
Seung-Bin Kim
claimed_verified
2023-11-22T13:11:26.639Z
655dfc0cf828e5ad0f1f833f
/avatars/2a36974ba91aead89b29a35265fe2398.svg
Seung-Bin Kim
user
sb-kim
655d90c9bf82497038e70805
Seong-Whan Lee
HierSpeech++: Bridging the Gap between Semantic and Acoustic
  Representation of Speech by Hierarchical Variational Inference for Zero-shot
  Speech Synthesis
Large language models (LLM)-based speech synthesis has been widely adopted in
zero-shot speech synthesis. However, they require a large-scale data and
possess the same limitations as previous autoregressive speech models,
including slow inference speed and lack of robustness. This paper proposes
HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech
(TTS) and voice conversion (VC). We verified that hierarchical speech synthesis
frameworks could significantly improve the robustness and expressiveness of the
synthetic speech. Furthermore, we significantly improve the naturalness and
speaker similarity of synthetic speech even in zero-shot speech synthesis
scenarios. For text-to-speech, we adopt the text-to-vec framework, which
generates a self-supervised speech representation and an F0 representation
based on text representations and prosody prompts. Then, HierSpeech++ generates
speech from the generated vector, F0, and voice prompt. We further introduce a
high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The
experimental results demonstrated that the hierarchical variational autoencoder
could be a strong zero-shot speech synthesizer given that it outperforms
LLM-based and diffusion-based models. Moreover, we achieved the first
human-level quality zero-shot speech synthesis. Audio samples and source code
are available at https://github.com/sh-lee-prml/HierSpeechpp.
HierSpeech++, an enhanced hierarchical variational autoencoder, achieves high-quality zero-shot speech synthesis by combining self-supervised representations and an efficient super-resolution framework.
large language models zero-shot speech synthesis hierarchical speech synthesis robustness expressiveness text-to-speech voice conversion text-to-vec self-supervised speech representation F0 representation prosody prompts speech super-resolution variational autoencoder human-level quality