BatonVoice: An Operationalist Framework for Enhancing Controllable
  Speech Synthesis with Linguistic Intelligence from LLMs
The rise of Large Language Models (LLMs) is reshaping multimodel models, with
speech synthesis being a prominent application. However, existing approaches
often underutilize the linguistic intelligence of these models, typically
failing to leverage their powerful instruction-following capabilities. This
limitation hinders the model's ability to follow text instructions for
controllable Text-to-Speech~(TTS). To address this, we propose a new paradigm
inspired by ``operationalism'' that decouples instruction understanding from
speech generation. We introduce BatonVoice, a framework where an LLM acts as a
``conductor'', understanding user instructions and generating a textual
``plan'' -- explicit vocal features (e.g., pitch, energy). A separate TTS
model, the ``orchestra'', then generates the speech from these features. To
realize this component, we develop BatonTTS, a TTS model trained specifically
for this task. Our experiments demonstrate that BatonVoice achieves strong
performance in controllable and emotional speech synthesis, outperforming
strong open- and closed-source baselines. Notably, our approach enables
remarkable zero-shot cross-lingual generalization, accurately applying feature
control abilities to languages unseen during post-training. This demonstrates
that objectifying speech into textual vocal features can more effectively
unlock the linguistic intelligence of LLMs.
68dc94d24159d1f2418f9b43
Yue Wang
claimed_verified
2025-10-05T12:47:43.763Z
641191c8756b9e455c7ef26d
/avatars/afc7a8a7a3ab0c5cae625e345f40eea5.svg
Yue Wang
user
Yue-Wang
68dc94d24159d1f2418f9b44
Ruotian Ma
68dc94d24159d1f2418f9b45
Xingyu Chen
68dc94d24159d1f2418f9b46
Zhengliang Shi
68dc94d24159d1f2418f9b47
Wanshun Chen
68dc94d24159d1f2418f9b48
Huang Liu
68dc94d24159d1f2418f9b49
Jiadi Yao
68dc94d24159d1f2418f9b4a
Qu Yang
68dc94d24159d1f2418f9b4b
Qingxuan Jiang
68dc94d24159d1f2418f9b4c
Fanghua Ye
68dc94d24159d1f2418f9b4d
Juntao Li
68dc94d24159d1f2418f9b4e
Min Zhang
68dc94d24159d1f2418f9b4f
Zhaopeng Tu
68dc94d24159d1f2418f9b50
Xiaolong Li
68dc94d24159d1f2418f9b51
Linus
BatonVoice: An Operationalist Framework for Enhancing Controllable
  Speech Synthesis with Linguistic Intelligence from LLMs
The rise of Large Language Models (LLMs) is reshaping multimodel models, with
speech synthesis being a prominent application. However, existing approaches
often underutilize the linguistic intelligence of these models, typically
failing to leverage their powerful instruction-following capabilities. This
limitation hinders the model's ability to follow text instructions for
controllable Text-to-Speech~(TTS). To address this, we propose a new paradigm
inspired by ``operationalism'' that decouples instruction understanding from
speech generation. We introduce BatonVoice, a framework where an LLM acts as a
``conductor'', understanding user instructions and generating a textual
``plan'' -- explicit vocal features (e.g., pitch, energy). A separate TTS
model, the ``orchestra'', then generates the speech from these features. To
realize this component, we develop BatonTTS, a TTS model trained specifically
for this task. Our experiments demonstrate that BatonVoice achieves strong
performance in controllable and emotional speech synthesis, outperforming
strong open- and closed-source baselines. Notably, our approach enables
remarkable zero-shot cross-lingual generalization, accurately applying feature
control abilities to languages unseen during post-training. This demonstrates
that objectifying speech into textual vocal features can more effectively
unlock the linguistic intelligence of LLMs.
BatonVoice framework decouples instruction understanding from speech generation, using an LLM to create vocal feature plans and a specialized TTS model to produce speech, achieving strong performance in controllable and emotional speech synthesis with zero-shot cross-lingual generalization.
Large Language Models multimodel models speech synthesis instruction-following capabilities Text-to-Speech operationalism BatonVoice BatonTTS vocal features controllable speech synthesis emotional speech synthesis zero-shot cross-lingual generalization