NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
  Diffusion Models
While recent large-scale text-to-speech (TTS) models have achieved
significant progress, they still fall short in speech quality, similarity, and
prosody. Considering speech intricately encompasses various attributes (e.g.,
content, prosody, timbre, and acoustic details) that pose significant
challenges for generation, a natural idea is to factorize speech into
individual subspaces representing different attributes and generate them
individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
novel factorized diffusion models to generate natural speech in a zero-shot
way. Specifically, 1) we design a neural codec with factorized vector
quantization (FVQ) to disentangle speech waveform into subspaces of content,
prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
model to generate attributes in each subspace following its corresponding
prompt. With this factorization design, NaturalSpeech 3 can effectively and
efficiently model the intricate speech with disentangled subspaces in a
divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
state-of-the-art TTS systems on quality, similarity, prosody, and
intelligibility. Furthermore, we achieve better performance by scaling to 1B
parameters and 200K hours of training data.
65e7f92674ab027493c2f417
Zeqian Ju
65e7f92674ab027493c2f418
Yuancheng Wang
claimed_verified
2025-08-27T07:13:42.757Z
63072d60cd148dbc5e49f4dd
/avatars/ffa61038c0ff20848fbcde7c1c34570e.svg
Yuancheng Wang
user
Hecheng0625
65e7f92674ab027493c2f419
Kai Shen
65e7f92674ab027493c2f41a
Xu Tan
extracted_confirmed
2024-04-05T05:51:26.298Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
65e7f92674ab027493c2f41b
Detai Xin
65e7f92674ab027493c2f41c
Dongchao Yang
65e7f92674ab027493c2f41d
Yanqing Liu
65e7f92674ab027493c2f41e
Yichong Leng
admin_assigned
2024-03-06T09:56:12.761Z
64a0347b528a9bbe59d6e08c
/avatars/6dd0bad84d711d1048a0a4169e621773.svg
Yichong Leng
user
ustcscallion
65e7f92674ab027493c2f41f
Kaitao Song
admin_assigned
2024-03-06T09:56:19.656Z
5fc0b2b61160c47d1d438568
/avatars/b355912b0ec683e73f21c8d36620e146.svg
Kaitao Song
user
KaitaoSong
65e7f92674ab027493c2f420
Siliang Tang
65e7f92674ab027493c2f421
Zhizheng Wu
claimed_verified
2024-07-05T06:56:14.830Z
63b4dcefa50cfcefdaa121f3
https://cdn-avatars.huggingface.co/v1/production/uploads/63b4dcefa50cfcefdaa121f3/MlIxOaTSCbZARVyo8Ly7r.jpeg
Dr Wuz
user
drwuz
65e7f92674ab027493c2f422
Tao Qin
65e7f92674ab027493c2f423
Xiang-Yang Li
65e7f92674ab027493c2f424
Wei Ye
65e7f92674ab027493c2f425
Shikun Zhang
65e7f92674ab027493c2f426
Jiang Bian
admin_assigned
2024-03-06T09:58:06.909Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
65e7f92674ab027493c2f427
Lei He
65e7f92674ab027493c2f428
Jinyu Li
65e7f92674ab027493c2f429
Sheng Zhao
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
  Diffusion Models
While recent large-scale text-to-speech (TTS) models have achieved
significant progress, they still fall short in speech quality, similarity, and
prosody. Considering speech intricately encompasses various attributes (e.g.,
content, prosody, timbre, and acoustic details) that pose significant
challenges for generation, a natural idea is to factorize speech into
individual subspaces representing different attributes and generate them
individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
novel factorized diffusion models to generate natural speech in a zero-shot
way. Specifically, 1) we design a neural codec with factorized vector
quantization (FVQ) to disentangle speech waveform into subspaces of content,
prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
model to generate attributes in each subspace following its corresponding
prompt. With this factorization design, NaturalSpeech 3 can effectively and
efficiently model the intricate speech with disentangled subspaces in a
divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
state-of-the-art TTS systems on quality, similarity, prosody, and
intelligibility. Furthermore, we achieve better performance by scaling to 1B
parameters and 200K hours of training data.
NaturalSpeech 3, using factorized diffusion models, outperforms other TTS systems by generating disentangled speech attributes in a zero-shot manner.
factorized diffusion models neural codec factorized vector quantization (FVQ) subspace content prosody timbre acoustic details text-to-speech (TTS) quality similarity intelligibility