LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive
  Modeling of Audio Discrete Codes
Prior works have demonstrated zero-shot text-to-speech by using a generative
language model on audio tokens obtained via a neural audio codec. It is still
challenging, however, to adapt them to low-latency scenarios. In this paper, we
present LiveSpeech - a fully autoregressive language model-based approach for
zero-shot text-to-speech, enabling low-latency streaming of the output audio.
To allow multiple token prediction within a single decoding step, we propose
(1) using adaptive codebook loss weights that consider codebook contribution in
each frame and focus on hard instances, and (2) grouping codebooks and
processing groups in parallel. Experiments show our proposed models achieve
competitive results to state-of-the-art baselines in terms of content accuracy,
speaker similarity, audio quality, and inference speed while being suitable for
low-latency streaming applications.
6661279f3902e37d2d90bc5c
Trung Dang
claimed_verified
2024-06-06T15:01:05.958Z
66046dc99fcaed55efc28bb2
/avatars/673bd6d1a5c6d58073b5feeea9c7c5e1.svg
Trung Dang
user
trungvd
6661279f3902e37d2d90bc5d
David Aponte
admin_assigned
2024-06-06T08:46:25.079Z
65b8003232a393895aff581d
/avatars/19dc5be2c790b7be73301a0beb9df94a.svg
David Aponte
user
bambamcito
6661279f3902e37d2d90bc5e
Dung Tran
claimed_verified
2024-06-09T20:05:31.467Z
6662197d69b8bbc79f5815b2
/avatars/7025332adbf6b6c0a9e9c92979f69584.svg
Dung Tran
user
dungntran
6661279f3902e37d2d90bc5f
Kazuhito Koishida
extracted_pending
2024-06-06T03:06:08.399Z
633f6bb8ac3954492880415f
/avatars/f3dd763d8e092f68cbc33469f148b337.svg
Kazuhito Koishida
user
koishida
LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive
  Modeling of Audio Discrete Codes
Prior works have demonstrated zero-shot text-to-speech by using a generative
language model on audio tokens obtained via a neural audio codec. It is still
challenging, however, to adapt them to low-latency scenarios. In this paper, we
present LiveSpeech - a fully autoregressive language model-based approach for
zero-shot text-to-speech, enabling low-latency streaming of the output audio.
To allow multiple token prediction within a single decoding step, we propose
(1) using adaptive codebook loss weights that consider codebook contribution in
each frame and focus on hard instances, and (2) grouping codebooks and
processing groups in parallel. Experiments show our proposed models achieve
competitive results to state-of-the-art baselines in terms of content accuracy,
speaker similarity, audio quality, and inference speed while being suitable for
low-latency streaming applications.
A novel autoregressive language model approach, LiveSpeech, enables low-latency text-to-speech by optimizing token prediction and parallel codebook processing.
generative language model audio tokens neural audio codec adaptive codebook loss weights autoregressive language model low-latency text-to-speech codebooks parallel processing content accuracy speaker similarity audio quality inference speed