CapSpeech: Enabling Downstream Applications in Style-Captioned
  Text-to-Speech
Recent advancements in generative artificial intelligence have significantly
transformed the field of style-captioned text-to-speech synthesis (CapTTS).
However, adapting CapTTS to real-world applications remains challenging due to
the lack of standardized, comprehensive datasets and limited research on
downstream tasks built upon CapTTS. To address these gaps, we introduce
CapSpeech, a new benchmark designed for a series of CapTTS-related tasks,
including style-captioned text-to-speech synthesis with sound events
(CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS
(EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech
comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36
million human-annotated audio-caption pairs. In addition, we introduce two new
datasets collected and recorded by a professional voice actor and experienced
audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside
the datasets, we conduct comprehensive experiments using both autoregressive
and non-autoregressive models on CapSpeech. Our results demonstrate
high-fidelity and highly intelligible speech synthesis across a diverse range
of speaking styles. To the best of our knowledge, CapSpeech is the largest
available dataset offering comprehensive annotations for CapTTS-related tasks.
The experiments and findings further provide valuable insights into the
challenges of developing CapTTS systems.
683fa315cd4314c4b3d53451
Helin Wang
claimed_verified
2025-06-06T07:42:05.814Z
63ecfb5ec5b3c734085db9ed
/avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
Helin Wang
user
westbrook
683fa315cd4314c4b3d53452
Jiarui Hai
claimed_verified
2025-06-06T07:42:08.194Z
6363e9e4969bdae89e10b5ac
https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
Jiarui Hai
user
Higobeatz
683fa315cd4314c4b3d53453
Dading Chong
683fa315cd4314c4b3d53454
Karan Thakkar
claimed_verified
2025-06-07T05:49:42.588Z
634452e00a284e1e9f1d3d9d
https://cdn-avatars.huggingface.co/v1/production/uploads/634452e00a284e1e9f1d3d9d/jlJXRNxg4OH-zT8z9oDrr.jpeg
Karan Thakkar
user
carankt
683fa315cd4314c4b3d53455
Tiantian Feng
claimed_verified
2025-06-11T08:37:02.607Z
64092a1ab6a334f53e278b3b
https://cdn-avatars.huggingface.co/v1/production/uploads/64092a1ab6a334f53e278b3b/tcueLWyyDL6WMUTw3Or4t.jpeg
Tiantian Feng
user
tiantiaf
683fa315cd4314c4b3d53456
Dongchao Yang
683fa315cd4314c4b3d53457
Junhyeok Lee
683fa315cd4314c4b3d53458
Laureano Moro Velazquez
683fa315cd4314c4b3d53459
Jesus Villalba
683fa315cd4314c4b3d5345a
Zengyi Qin
683fa315cd4314c4b3d5345b
Shrikanth Narayanan
683fa315cd4314c4b3d5345c
Mounya Elhiali
683fa315cd4314c4b3d5345d
Najim Dehak
CapSpeech: Enabling Downstream Applications in Style-Captioned
  Text-to-Speech
Recent advancements in generative artificial intelligence have significantly
transformed the field of style-captioned text-to-speech synthesis (CapTTS).
However, adapting CapTTS to real-world applications remains challenging due to
the lack of standardized, comprehensive datasets and limited research on
downstream tasks built upon CapTTS. To address these gaps, we introduce
CapSpeech, a new benchmark designed for a series of CapTTS-related tasks,
including style-captioned text-to-speech synthesis with sound events
(CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS
(EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech
comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36
million human-annotated audio-caption pairs. In addition, we introduce two new
datasets collected and recorded by a professional voice actor and experienced
audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside
the datasets, we conduct comprehensive experiments using both autoregressive
and non-autoregressive models on CapSpeech. Our results demonstrate
high-fidelity and highly intelligible speech synthesis across a diverse range
of speaking styles. To the best of our knowledge, CapSpeech is the largest
available dataset offering comprehensive annotations for CapTTS-related tasks.
The experiments and findings further provide valuable insights into the
challenges of developing CapTTS systems.
CapSpeech introduces a large benchmark dataset for various captioned text-to-speech tasks, facilitating advancements in style, accent, emotion, and chat-agent synthesis.
autoregressive models non-autoregressive models CapTTS CapTTS-SE AccCapTTS EmoCapTTS AgentTTS machine-annotated human-annotated