TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language
  Modeling
Speech tokenizers serve as foundational components for speech language
models, yet current designs exhibit several limitations, including: 1)
dependence on multi-layer residual vector quantization structures or high frame
rates, 2) reliance on auxiliary pre-trained models for semantic distillation,
and 3) requirements for complex two-stage training processes. In this work, we
introduce the Text-aware Diffusion Transformer Speech Codec (TaDiCodec), a
novel approach designed to overcome these challenges. TaDiCodec employs
end-to-end optimization for quantization and reconstruction through a diffusion
autoencoder, while integrating text guidance into the diffusion decoder to
enhance reconstruction quality and achieve optimal compression. TaDiCodec
achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of
0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining
superior performance on critical speech generation evaluation metrics such as
Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS).
Notably, TaDiCodec employs a single-stage, end-to-end training paradigm, and
obviating the need for auxiliary pre-trained models. We also validate the
compatibility of TaDiCodec in language model based zero-shot text-to-speech
with both autoregressive modeling and masked generative modeling, demonstrating
its effectiveness and efficiency for speech language modeling, as well as a
significantly small reconstruction-generation gap. We will open source our code
and model checkpoints. Audio samples are are available at
https:/tadicodec.github.io/. We release code and model checkpoints at
https:/github.com/HeCheng0625/Diffusion-Speech-Tokenizer.
68ad0ef486b21a0e2e358d05
Yuancheng Wang
claimed_verified
2025-08-27T07:11:26.295Z
63072d60cd148dbc5e49f4dd
/avatars/ffa61038c0ff20848fbcde7c1c34570e.svg
Yuancheng Wang
user
Hecheng0625
68ad0ef486b21a0e2e358d06
Dekun Chen
68ad0ef486b21a0e2e358d07
Xueyao Zhang
68ad0ef486b21a0e2e358d08
Junan Zhang
claimed_verified
2025-09-12T16:22:08.121Z
63f8979cb0ae1748524be22c
https://cdn-avatars.huggingface.co/v1/production/uploads/1677236096253-noauth.png
junan zhang
user
viewfinder-annn
68ad0ef486b21a0e2e358d09
Jiaqi Li
68ad0ef486b21a0e2e358d0a
Zhizheng Wu
TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language
  Modeling
Speech tokenizers serve as foundational components for speech language
models, yet current designs exhibit several limitations, including: 1)
dependence on multi-layer residual vector quantization structures or high frame
rates, 2) reliance on auxiliary pre-trained models for semantic distillation,
and 3) requirements for complex two-stage training processes. In this work, we
introduce the Text-aware Diffusion Transformer Speech Codec (TaDiCodec), a
novel approach designed to overcome these challenges. TaDiCodec employs
end-to-end optimization for quantization and reconstruction through a diffusion
autoencoder, while integrating text guidance into the diffusion decoder to
enhance reconstruction quality and achieve optimal compression. TaDiCodec
achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of
0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining
superior performance on critical speech generation evaluation metrics such as
Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS).
Notably, TaDiCodec employs a single-stage, end-to-end training paradigm, and
obviating the need for auxiliary pre-trained models. We also validate the
compatibility of TaDiCodec in language model based zero-shot text-to-speech
with both autoregressive modeling and masked generative modeling, demonstrating
its effectiveness and efficiency for speech language modeling, as well as a
significantly small reconstruction-generation gap. We will open source our code
and model checkpoints. Audio samples are are available at
https:/tadicodec.github.io/. We release code and model checkpoints at
https:/github.com/HeCheng0625/Diffusion-Speech-Tokenizer.
TaDiCodec, a Text-aware Diffusion Transformer Speech Codec, achieves low frame rates and bitrates with superior speech generation performance using end-to-end optimization and text guidance.
diffusion autoencoder text guidance end-to-end optimization Word Error Rate (WER) speaker similarity (SIM) speech quality (UTMOS) zero-shot text-to-speech autoregressive modeling masked generative modeling