Boosting Large Language Model for Speech Synthesis: An Empirical Study
Large language models (LLMs) have made significant advancements in natural
language processing and are concurrently extending the language ability to
other modalities, such as speech and vision. Nevertheless, most of the previous
work focuses on prompting LLMs with perception abilities like auditory
comprehension, and the effective approach for augmenting LLMs with speech
synthesis capabilities remains ambiguous. In this paper, we conduct a
comprehensive empirical exploration of boosting LLMs with the ability to
generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
synthesis model VALL-E. We compare three integration methods between LLMs and
speech synthesis models, including directly fine-tuned LLMs, superposed layers
of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
encoder. Experimental results show that, using LoRA method to fine-tune LLMs
directly to boost the speech synthesis capability does not work well, and
superposed LLMs and VALL-E can improve the quality of generated speech both in
speaker similarity and word error rate (WER). Among these three methods,
coupled methods leveraging LLMs as the text encoder can achieve the best
performance, making it outperform original speech synthesis models with a
consistently better speaker similarity and a significant (10.9%) WER reduction.
65939ab054f8826173507d00
Hongkun Hao
65939ab054f8826173507d01
Long Zhou
65939ab054f8826173507d02
Shujie Liu
65939ab054f8826173507d03
Jinyu Li
65939ab054f8826173507d04
Shujie Hu
65939ab054f8826173507d05
Rui Wang
65939ab054f8826173507d06
Furu Wei
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Large language models (LLMs) have made significant advancements in natural
language processing and are concurrently extending the language ability to
other modalities, such as speech and vision. Nevertheless, most of the previous
work focuses on prompting LLMs with perception abilities like auditory
comprehension, and the effective approach for augmenting LLMs with speech
synthesis capabilities remains ambiguous. In this paper, we conduct a
comprehensive empirical exploration of boosting LLMs with the ability to
generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
synthesis model VALL-E. We compare three integration methods between LLMs and
speech synthesis models, including directly fine-tuned LLMs, superposed layers
of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
encoder. Experimental results show that, using LoRA method to fine-tune LLMs
directly to boost the speech synthesis capability does not work well, and
superposed LLMs and VALL-E can improve the quality of generated speech both in
speaker similarity and word error rate (WER). Among these three methods,
coupled methods leveraging LLMs as the text encoder can achieve the best
performance, making it outperform original speech synthesis models with a
consistently better speaker similarity and a significant (10.9%) WER reduction.
Coupling large language models with text-to-speech synthesis by using the LLM as a text encoder significantly improves speech quality and reduces word error rate.
large language models LLMs natural language processing speech synthesis perception abilities auditory comprehension pre-trained LLMs LLaMA/OPT text-to-speech synthesis model VALL-E fine-tuning superposed layers LoRA method speaker similarity word error rate WER