VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text
  to Speech Synthesizers
This paper introduces VALL-E 2, the latest advancement in neural codec
language models that marks a milestone in zero-shot text-to-speech synthesis
(TTS), achieving human parity for the first time. Based on its predecessor,
VALL-E, the new iteration introduces two significant enhancements: Repetition
Aware Sampling refines the original nucleus sampling process by accounting for
token repetition in the decoding history. It not only stabilizes the decoding
but also circumvents the infinite loop issue. Grouped Code Modeling organizes
codec codes into groups to effectively shorten the sequence length, which not
only boosts inference speed but also addresses the challenges of long sequence
modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
2 surpasses previous systems in speech robustness, naturalness, and speaker
similarity. It is the first of its kind to reach human parity on these
benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
even for sentences that are traditionally challenging due to their complexity
or repetitive phrases. The advantages of this work could contribute to valuable
endeavors, such as generating speech for individuals with aphasia or people
with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
https://aka.ms/valle2.
6667c3d28b7703d4b30a5d32
Sanyuan Chen
admin_assigned
2024-06-11T10:16:32.142Z
6465843ce27766e8921b9cab
/avatars/6017db24da593c9afb21edfcedcddfd4.svg
Sanyuan Chen
user
Sanyuan-Chen
6667c3d28b7703d4b30a5d33
Shujie Liu
admin_assigned
2024-06-11T10:16:38.358Z
64dc94f1bdac91d86834e13b
/avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
Shujie Liu
user
j4ckl1u
6667c3d28b7703d4b30a5d34
Long Zhou
6667c3d28b7703d4b30a5d35
Yanqing Liu
admin_assigned
2024-06-11T10:26:54.652Z
654a771a2d2fcd6bf27ff0de
/avatars/6312b23c642009f13c7993ce4658d06d.svg
Liuyanqing
user
Yanqing0327
6667c3d28b7703d4b30a5d36
Xu Tan
admin_assigned
2024-06-11T10:27:07.844Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
6667c3d28b7703d4b30a5d37
Jinyu Li
admin_assigned
2024-06-11T10:27:30.230Z
644dffbe16703fd6702d3bd5
/avatars/dab5831f689ffec9cd1f63213539d3f0.svg
Jinyu Li
user
JinyuID
6667c3d28b7703d4b30a5d38
Sheng Zhao
6667c3d28b7703d4b30a5d39
Yao Qian
6667c3d28b7703d4b30a5d3a
Furu Wei
admin_assigned
2024-06-11T10:25:59.688Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text
  to Speech Synthesizers
This paper introduces VALL-E 2, the latest advancement in neural codec
language models that marks a milestone in zero-shot text-to-speech synthesis
(TTS), achieving human parity for the first time. Based on its predecessor,
VALL-E, the new iteration introduces two significant enhancements: Repetition
Aware Sampling refines the original nucleus sampling process by accounting for
token repetition in the decoding history. It not only stabilizes the decoding
but also circumvents the infinite loop issue. Grouped Code Modeling organizes
codec codes into groups to effectively shorten the sequence length, which not
only boosts inference speed but also addresses the challenges of long sequence
modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
2 surpasses previous systems in speech robustness, naturalness, and speaker
similarity. It is the first of its kind to reach human parity on these
benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
even for sentences that are traditionally challenging due to their complexity
or repetitive phrases. The advantages of this work could contribute to valuable
endeavors, such as generating speech for individuals with aphasia or people
with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
https://aka.ms/valle2.
VALL-E 2 achieves human parity in zero-shot text-to-speech synthesis through improvements in sampling and code modeling.
neural codec language models zero-shot text-to-speech synthesis (TTS) human parity Repetition Aware Sampling nucleus sampling Grouped Code Modeling speech robustness naturalness speaker similarity LibriSpeech VCTK datasets