FlashSpeech: Efficient Zero-Shot Speech Synthesis
Recent progress in large-scale zero-shot speech synthesis has been
significantly advanced by language models and diffusion models. However, the
generation process of both methods is slow and computationally intensive.
Efficient speech synthesis using a lower computing budget to achieve quality on
par with previous work remains a significant challenge. In this paper, we
present FlashSpeech, a large-scale zero-shot speech synthesis system with
approximately 5\% of the inference time compared with previous work.
FlashSpeech is built on the latent consistency model and applies a novel
adversarial consistency training approach that can train from scratch without
the need for a pre-trained diffusion model as the teacher. Furthermore, a new
prosody generator module enhances the diversity of prosody, making the rhythm
of the speech sound more natural. The generation processes of FlashSpeech can
be achieved efficiently with one or two sampling steps while maintaining high
audio quality and high similarity to the audio prompt for zero-shot speech
generation. Our experimental results demonstrate the superior performance of
FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other
zero-shot speech synthesis systems while maintaining comparable performance in
terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates
its versatility by efficiently performing tasks like voice conversion, speech
editing, and diverse speech sampling. Audio samples can be found in
https://flashspeech.github.io/.
662868cc189fe0763bb8a2fe
Zhen Ye
claimed_verified
2024-12-06T15:58:36.245Z
645f172d7c6bff8577353d1a
/avatars/a83682e1343809257b082b78d58c582a.svg
ZhenYE
user
ZhenYe234
662868cc189fe0763bb8a2ff
Zeqian Ju
662868cc189fe0763bb8a300
Haohe Liu
admin_assigned
2024-04-24T12:05:52.736Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
662868cc189fe0763bb8a301
Xu Tan
admin_assigned
2024-04-24T12:06:12.115Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
662868cc189fe0763bb8a302
Jianyi Chen
662868cc189fe0763bb8a303
Yiwen Lu
662868cc189fe0763bb8a304
Peiwen Sun
662868cc189fe0763bb8a305
Jiahao Pan
admin_assigned
2025-03-13T15:40:04.783Z
647f7cf4b0e9676458a014f9
https://cdn-avatars.huggingface.co/v1/production/uploads/647f7cf4b0e9676458a014f9/4L_0MbVErSmPYrk3mIzNP.jpeg
jiahao
user
cherishpjh
662868cc189fe0763bb8a306
Weizhen Bian
662868cc189fe0763bb8a307
Shulin He
662868cc189fe0763bb8a308
Qifeng Liu
662868cc189fe0763bb8a309
Yike Guo
admin_assigned
2024-04-24T12:07:49.056Z
64762b325f70f9b2d0ade28e
/avatars/1f37382724b475ece805f943a8858acd.svg
Yike Guo
user
SuaLily
662868cc189fe0763bb8a30a
Wei Xue
claimed_verified
2024-04-24T11:11:21.136Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
FlashSpeech: Efficient Zero-Shot Speech Synthesis
Recent progress in large-scale zero-shot speech synthesis has been
significantly advanced by language models and diffusion models. However, the
generation process of both methods is slow and computationally intensive.
Efficient speech synthesis using a lower computing budget to achieve quality on
par with previous work remains a significant challenge. In this paper, we
present FlashSpeech, a large-scale zero-shot speech synthesis system with
approximately 5\% of the inference time compared with previous work.
FlashSpeech is built on the latent consistency model and applies a novel
adversarial consistency training approach that can train from scratch without
the need for a pre-trained diffusion model as the teacher. Furthermore, a new
prosody generator module enhances the diversity of prosody, making the rhythm
of the speech sound more natural. The generation processes of FlashSpeech can
be achieved efficiently with one or two sampling steps while maintaining high
audio quality and high similarity to the audio prompt for zero-shot speech
generation. Our experimental results demonstrate the superior performance of
FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other
zero-shot speech synthesis systems while maintaining comparable performance in
terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates
its versatility by efficiently performing tasks like voice conversion, speech
editing, and diverse speech sampling. Audio samples can be found in
https://flashspeech.github.io/.
FlashSpeech, using latent consistency model and adversarial consistency training, achieves fast and high-quality zero-shot speech synthesis with enhanced prosody generation.
latent consistency model adversarial consistency training zero-shot speech synthesis prosody generator