NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech
  Modeling with Paralinguistic Vocalizations
Paralinguistic vocalizations-including non-verbal sounds like laughter and
breathing, as well as lexicalized interjections such as "uhm" and "oh"-are
integral to natural spoken communication. Despite their importance in conveying
affect, intent, and interactional cues, such cues remain largely overlooked in
conventional automatic speech recognition (ASR) and text-to-speech (TTS)
systems. We present NVSpeech, an integrated and scalable pipeline that bridges
the recognition and synthesis of paralinguistic vocalizations, encompassing
dataset construction, ASR modeling, and controllable TTS. (1) We introduce a
manually annotated dataset of 48,430 human-spoken utterances with 18 word-level
paralinguistic categories. (2) We develop the paralinguistic-aware ASR model,
which treats paralinguistic cues as inline decodable tokens (e.g., "You're so
funny [Laughter]"), enabling joint lexical and non-verbal transcription. This
model is then used to automatically annotate a large corpus, the first
large-scale Chinese dataset of 174,179 utterances (573 hours) with word-level
alignment and paralingustic cues. (3) We finetune zero-shot TTS models on both
human- and auto-labeled data to enable explicit control over paralinguistic
vocalizations, allowing context-aware insertion at arbitrary token positions
for human-like speech synthesis. By unifying the recognition and generation of
paralinguistic vocalizations, NVSpeech offers the first open, large-scale,
word-level annotated pipeline for expressive speech modeling in Mandarin,
integrating recognition and synthesis in a scalable and controllable manner.
Dataset and audio demos are available at https://nvspeech170k.github.io/.
689c33fbfab6fdd2e52ac8ec
Huan Liao
689c33fbfab6fdd2e52ac8ed
Qinke Ni
689c33fbfab6fdd2e52ac8ee
Yuancheng Wang
claimed_verified
2025-08-27T07:11:54.105Z
63072d60cd148dbc5e49f4dd
/avatars/ffa61038c0ff20848fbcde7c1c34570e.svg
Yuancheng Wang
user
Hecheng0625
689c33fbfab6fdd2e52ac8ef
Yiheng Lu
689c33fbfab6fdd2e52ac8f0
Haoyue Zhan
689c33fbfab6fdd2e52ac8f1
Pengyuan Xie
689c33fbfab6fdd2e52ac8f2
Qiang Zhang
689c33fbfab6fdd2e52ac8f3
Zhizheng Wu
NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech
  Modeling with Paralinguistic Vocalizations
Paralinguistic vocalizations-including non-verbal sounds like laughter and
breathing, as well as lexicalized interjections such as "uhm" and "oh"-are
integral to natural spoken communication. Despite their importance in conveying
affect, intent, and interactional cues, such cues remain largely overlooked in
conventional automatic speech recognition (ASR) and text-to-speech (TTS)
systems. We present NVSpeech, an integrated and scalable pipeline that bridges
the recognition and synthesis of paralinguistic vocalizations, encompassing
dataset construction, ASR modeling, and controllable TTS. (1) We introduce a
manually annotated dataset of 48,430 human-spoken utterances with 18 word-level
paralinguistic categories. (2) We develop the paralinguistic-aware ASR model,
which treats paralinguistic cues as inline decodable tokens (e.g., "You're so
funny [Laughter]"), enabling joint lexical and non-verbal transcription. This
model is then used to automatically annotate a large corpus, the first
large-scale Chinese dataset of 174,179 utterances (573 hours) with word-level
alignment and paralingustic cues. (3) We finetune zero-shot TTS models on both
human- and auto-labeled data to enable explicit control over paralinguistic
vocalizations, allowing context-aware insertion at arbitrary token positions
for human-like speech synthesis. By unifying the recognition and generation of
paralinguistic vocalizations, NVSpeech offers the first open, large-scale,
word-level annotated pipeline for expressive speech modeling in Mandarin,
integrating recognition and synthesis in a scalable and controllable manner.
Dataset and audio demos are available at https://nvspeech170k.github.io/.
NVSpeech is a pipeline that integrates the recognition and synthesis of paralinguistic vocalizations in Mandarin, using a large annotated dataset and models that treat these cues as decodable tokens.
paralinguistic vocalizations ASR modeling TTS paralinguistic-aware ASR zero-shot TTS word-level paralinguistic categories paralinguistic cues lexical and non-verbal transcription human-like speech synthesis expressive speech modeling