Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
  Speech Synthesis
Recent advances in text-based large language models (LLMs), particularly in
the GPT series and the o1 model, have demonstrated the effectiveness of scaling
both training-time and inference-time compute. However, current
state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
separate models (e.g., diffusion models after LLM), complicating the decision
of whether to scale a particular model during training or testing. This work
makes the following contributions: First, we explore the scaling of train-time
and inference-time compute for speech synthesis. Second, we propose a simple
framework Llasa for speech synthesis that employs a single-layer vector
quantizer (VQ) codec and a single Transformer architecture to fully align with
standard LLMs such as Llama. Our experiments reveal that scaling train-time
compute for Llasa consistently improves the naturalness of synthesized speech
and enables the generation of more complex and accurate prosody patterns.
Furthermore, from the perspective of scaling inference-time compute, we employ
speech understanding models as verifiers during the search, finding that
scaling inference-time compute shifts the sampling modes toward the preferences
of specific verifiers, thereby improving emotional expressiveness, timbre
consistency, and content accuracy. In addition, we released the checkpoint and
training code for our TTS model (1B, 3B, 8B) and codec model publicly
available.
67a5894db16fabcdd2de5459
Zhen Ye
claimed_verified
2025-02-07T09:58:08.787Z
645f172d7c6bff8577353d1a
/avatars/a83682e1343809257b082b78d58c582a.svg
ZhenYE
user
ZhenYe234
67a5894db16fabcdd2de545a
Xinfa Zhu
67a5894db16fabcdd2de545b
Chi-Min Chan
67a5894db16fabcdd2de545c
Xinsheng Wang
67a5894db16fabcdd2de545d
Xu Tan
claimed_verified
2025-03-12T08:45:27.312Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
67a5894db16fabcdd2de545e
Jiahe Lei
67a5894db16fabcdd2de545f
Yi Peng
67a5894db16fabcdd2de5460
Haohe Liu
67a5894db16fabcdd2de5461
Yizhu Jin
67a5894db16fabcdd2de5462
Zheqi DAI
67a5894db16fabcdd2de5463
Hongzhan Lin
claimed_verified
2025-02-19T09:05:09.037Z
6499466c7d1edf7cb612a9a6
/avatars/c2e18594aa0879db8226f2a04496fb0b.svg
Hongzhan Lin
user
danielhzlin
67a5894db16fabcdd2de5464
Jianyi Chen
67a5894db16fabcdd2de5465
Xingjian Du
67a5894db16fabcdd2de5466
Liumeng Xue
67a5894db16fabcdd2de5467
Yunlin Chen
67a5894db16fabcdd2de5468
Zhifei Li
67a5894db16fabcdd2de5469
Lei Xie
67a5894db16fabcdd2de546a
Qiuqiang Kong
67a5894db16fabcdd2de546b
Yike Guo
67a5894db16fabcdd2de546c
Wei Xue
extracted_pending
2025-02-07T04:17:17.888Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
  Speech Synthesis
Recent advances in text-based large language models (LLMs), particularly in
the GPT series and the o1 model, have demonstrated the effectiveness of scaling
both training-time and inference-time compute. However, current
state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
separate models (e.g., diffusion models after LLM), complicating the decision
of whether to scale a particular model during training or testing. This work
makes the following contributions: First, we explore the scaling of train-time
and inference-time compute for speech synthesis. Second, we propose a simple
framework Llasa for speech synthesis that employs a single-layer vector
quantizer (VQ) codec and a single Transformer architecture to fully align with
standard LLMs such as Llama. Our experiments reveal that scaling train-time
compute for Llasa consistently improves the naturalness of synthesized speech
and enables the generation of more complex and accurate prosody patterns.
Furthermore, from the perspective of scaling inference-time compute, we employ
speech understanding models as verifiers during the search, finding that
scaling inference-time compute shifts the sampling modes toward the preferences
of specific verifiers, thereby improving emotional expressiveness, timbre
consistency, and content accuracy. In addition, we released the checkpoint and
training code for our TTS model (1B, 3B, 8B) and codec model publicly
available.
A single-layer vector quantizer codec and Transformer-based framework for text-to-speech synthesis improves speech naturalness and emotional expressiveness with scalable compute resources.
large language models GPT series diffusion models speech synthesis vector quantizer (VQ) codec Transformer architecture train-time compute inference-time compute prosody patterns speech understanding models verifiers sampling modes emotional expressiveness timbre consistency content accuracy