FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs
Speech tokenization enables discrete representation and facilitates speech
language modeling. However, existing neural codecs capture low-level acoustic
features, overlooking the semantic and contextual cues inherent to human
speech. While recent efforts introduced semantic representations from
self-supervised speech models or incorporated contextual representations from
pre-trained language models, challenges remain in aligning and unifying the
semantic and contextual representations. We introduce FuseCodec, which unifies
acoustic, semantic, and contextual representations through strong cross-modal
alignment and globally informed supervision. We propose three complementary
techniques: (i) Latent Representation Fusion, integrating semantic and
contextual features directly into the encoder latent space for robust and
unified representation learning; (ii) Global Semantic-Contextual Supervision,
supervising discrete tokens with globally pooled and broadcasted
representations to enhance temporal consistency and cross-modal alignment; and
(iii) Temporally Aligned Contextual Supervision, strengthening alignment by
dynamically matching contextual and speech tokens within a local window for
fine-grained token-level supervision. We further introduce FuseCodec-TTS,
demonstrating our methodology's applicability to zero-shot speech synthesis.
Empirically, FuseCodec achieves state-of-the-art performance in LibriSpeech,
surpassing EnCodec, SpeechTokenizer, and DAC in transcription accuracy,
perceptual quality, intelligibility, and speaker similarity. Results highlight
the effectiveness of contextually and semantically guided tokenization for
speech tokenization and downstream tasks. Code and pretrained models are
available at https://github.com/mubtasimahasan/FuseCodec.
68ca01d66e0073c09bd1ddc5
Md Mubtasim Ahasan
68ca01d66e0073c09bd1ddc6
Rafat Hasan Khan
68ca01d66e0073c09bd1ddc7
Tasnim Mohiuddin
68ca01d66e0073c09bd1ddc8
Aman Chadha
claimed_verified
2025-09-17T12:57:15.390Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
68ca01d66e0073c09bd1ddc9
Tariq Iqbal
68ca01d66e0073c09bd1ddca
M Ashraful Amin
68ca01d66e0073c09bd1ddcb
Amin Ahsan Ali
68ca01d66e0073c09bd1ddcc
Md Mofijul Islam
68ca01d66e0073c09bd1ddcd
A K M Mahbubur Rahman
FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs
Speech tokenization enables discrete representation and facilitates speech
language modeling. However, existing neural codecs capture low-level acoustic
features, overlooking the semantic and contextual cues inherent to human
speech. While recent efforts introduced semantic representations from
self-supervised speech models or incorporated contextual representations from
pre-trained language models, challenges remain in aligning and unifying the
semantic and contextual representations. We introduce FuseCodec, which unifies
acoustic, semantic, and contextual representations through strong cross-modal
alignment and globally informed supervision. We propose three complementary
techniques: (i) Latent Representation Fusion, integrating semantic and
contextual features directly into the encoder latent space for robust and
unified representation learning; (ii) Global Semantic-Contextual Supervision,
supervising discrete tokens with globally pooled and broadcasted
representations to enhance temporal consistency and cross-modal alignment; and
(iii) Temporally Aligned Contextual Supervision, strengthening alignment by
dynamically matching contextual and speech tokens within a local window for
fine-grained token-level supervision. We further introduce FuseCodec-TTS,
demonstrating our methodology's applicability to zero-shot speech synthesis.
Empirically, FuseCodec achieves state-of-the-art performance in LibriSpeech,
surpassing EnCodec, SpeechTokenizer, and DAC in transcription accuracy,
perceptual quality, intelligibility, and speaker similarity. Results highlight
the effectiveness of contextually and semantically guided tokenization for
speech tokenization and downstream tasks. Code and pretrained models are
available at https://github.com/mubtasimahasan/FuseCodec.
FuseCodec unifies acoustic, semantic, and contextual representations in speech tokenization through cross-modal alignment and global supervision, achieving state-of-the-art performance in transcription and synthesis.
speech tokenization discrete representation speech language modeling neural codecs acoustic features semantic representations self-supervised speech models contextual representations pre-trained language models cross-modal alignment globally informed supervision Latent Representation Fusion Global Semantic-Contextual Supervision Temporally Aligned Contextual Supervision FuseCodec-TTS LibriSpeech EnCodec SpeechTokenizer DAC transcription accuracy perceptual quality intelligibility speaker similarity