CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency
  Model
Denoising diffusion probabilistic models (DDPMs) have shown promising
performance for speech synthesis. However, a large number of iterative steps
are required to achieve high sample quality, which restricts the inference
speed. Maintaining sample quality while increasing sampling speed has become a
challenging task. In this paper, we propose a "Co"nsistency "Mo"del-based
"Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a
single diffusion sampling step while achieving high audio quality. The
consistency constraint is applied to distill a consistency model from a
well-designed diffusion-based teacher model, which ultimately yields superior
performances in the distilled CoMoSpeech. Our experiments show that by
generating audio recordings by a single sampling step, the CoMoSpeech achieves
an inference speed more than 150 times faster than real-time on a single NVIDIA
A100 GPU, which is comparable to FastSpeech2, making diffusion-sampling based
speech synthesis truly practical. Meanwhile, objective and subjective
evaluations on text-to-speech and singing voice synthesis show that the
proposed teacher models yield the best audio quality, and the one-step sampling
based CoMoSpeech achieves the best inference speed with better or comparable
audio quality to other conventional multi-step diffusion model baselines. Audio
samples are available at https://comospeech.github.io/.
645d9731f4f49de580a07751
Zhen Ye
claimed_verified
2024-12-06T15:59:15.666Z
645f172d7c6bff8577353d1a
/avatars/a83682e1343809257b082b78d58c582a.svg
ZhenYE
user
ZhenYe234
645d9731f4f49de580a07752
Wei Xue
claimed_verified
2024-04-24T11:14:28.337Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
645d9731f4f49de580a07753
Xu Tan
claimed_verified
2024-04-05T08:39:45.256Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
645d9731f4f49de580a07754
Jie Chen
645d9731f4f49de580a07755
Qifeng Liu
645d9731f4f49de580a07756
Yike Guo
admin_assigned
2023-06-02T14:56:05.843Z
64762b325f70f9b2d0ade28e
/avatars/1f37382724b475ece805f943a8858acd.svg
Yike Guo
user
SuaLily
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency
  Model
Denoising diffusion probabilistic models (DDPMs) have shown promising
performance for speech synthesis. However, a large number of iterative steps
are required to achieve high sample quality, which restricts the inference
speed. Maintaining sample quality while increasing sampling speed has become a
challenging task. In this paper, we propose a "Co"nsistency "Mo"del-based
"Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a
single diffusion sampling step while achieving high audio quality. The
consistency constraint is applied to distill a consistency model from a
well-designed diffusion-based teacher model, which ultimately yields superior
performances in the distilled CoMoSpeech. Our experiments show that by
generating audio recordings by a single sampling step, the CoMoSpeech achieves
an inference speed more than 150 times faster than real-time on a single NVIDIA
A100 GPU, which is comparable to FastSpeech2, making diffusion-sampling based
speech synthesis truly practical. Meanwhile, objective and subjective
evaluations on text-to-speech and singing voice synthesis show that the
proposed teacher models yield the best audio quality, and the one-step sampling
based CoMoSpeech achieves the best inference speed with better or comparable
audio quality to other conventional multi-step diffusion model baselines. Audio
samples are available at https://comospeech.github.io/.
CoMoSpeech, a consistency model-based method, achieves high-quality speech synthesis with a single diffusion sampling step, significantly speeding up inference compared to traditional models.
denoising diffusion probabilistic models DDPMs speech synthesis consistency model diffusion-based teacher model text-to-speech singing voice synthesis