RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting
  for Text-to-Speech Synthesis
We present RALL-E, a robust language modeling method for text-to-speech (TTS)
synthesis. While previous work based on large language models (LLMs) shows
impressive performance on zero-shot TTS, such methods often suffer from poor
robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
high word error rate (WER), due to the autoregressive prediction style of
language models. The core idea behind RALL-E is chain-of-thought (CoT)
prompting, which decomposes the task into simpler steps to enhance the
robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
prosody features (pitch and duration) of the input text and uses them as
intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E
utilizes the predicted duration prompt to guide the computing of self-attention
weights in Transformer to enforce the model to focus on the corresponding
phonemes and prosody features when predicting speech tokens. Results of
comprehensive objective and subjective evaluations demonstrate that, compared
to a powerful baseline method VALL-E, RALL-E significantly improves the WER of
zero-shot TTS from 6.3% (without reranking) and 2.1% (with reranking) to
2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E
correctly synthesizes sentences that are hard for VALL-E and reduces the error
rate from 68% to 4%.
660f69515a39fcf64158ed69
Detai Xin
660f69515a39fcf64158ed6a
Xu Tan
extracted_confirmed
2024-04-05T05:51:36.839Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
660f69515a39fcf64158ed6b
Kai Shen
660f69515a39fcf64158ed6c
Zeqian Ju
660f69515a39fcf64158ed6d
Dongchao Yang
660f69515a39fcf64158ed6e
Yuancheng Wang
claimed_verified
2025-08-27T07:12:48.899Z
63072d60cd148dbc5e49f4dd
/avatars/ffa61038c0ff20848fbcde7c1c34570e.svg
Yuancheng Wang
user
Hecheng0625
660f69515a39fcf64158ed6f
Shinnosuke Takamichi
660f69515a39fcf64158ed70
Hiroshi Saruwatari
660f69515a39fcf64158ed71
Shujie Liu
660f69515a39fcf64158ed72
Jinyu Li
660f69515a39fcf64158ed73
Sheng Zhao
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting
  for Text-to-Speech Synthesis
We present RALL-E, a robust language modeling method for text-to-speech (TTS)
synthesis. While previous work based on large language models (LLMs) shows
impressive performance on zero-shot TTS, such methods often suffer from poor
robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
high word error rate (WER), due to the autoregressive prediction style of
language models. The core idea behind RALL-E is chain-of-thought (CoT)
prompting, which decomposes the task into simpler steps to enhance the
robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
prosody features (pitch and duration) of the input text and uses them as
intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E
utilizes the predicted duration prompt to guide the computing of self-attention
weights in Transformer to enforce the model to focus on the corresponding
phonemes and prosody features when predicting speech tokens. Results of
comprehensive objective and subjective evaluations demonstrate that, compared
to a powerful baseline method VALL-E, RALL-E significantly improves the WER of
zero-shot TTS from 6.3% (without reranking) and 2.1% (with reranking) to
2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E
correctly synthesizes sentences that are hard for VALL-E and reduces the error
rate from 68% to 4%.
RALL-E, a robust text-to-speech method using chain-of-thought prompting, enhances prosody prediction and reduces word error rate in zero-shot TTS compared to VALL-E.
large language models zero-shot TTS autoregressive prediction chain-of-thought prompting prosody features pitch duration speech tokens self-attention Transformer WER reranking