Discrete Audio Tokens: More Than a Survey!
Discrete audio tokens are compact representations that aim to preserve
perceptual quality, phonetic content, and speaker characteristics while
enabling efficient storage and inference, as well as competitive performance
across diverse downstream tasks.They provide a practical alternative to
continuous features, enabling the integration of speech and audio into modern
large language models (LLMs). As interest in token-based audio processing
grows, various tokenization methods have emerged, and several surveys have
reviewed the latest progress in the field. However, existing studies often
focus on specific domains or tasks and lack a unified comparison across various
benchmarks. This paper presents a systematic review and benchmark of discrete
audio tokenizers, covering three domains: speech, music, and general audio. We
propose a taxonomy of tokenization approaches based on encoder-decoder,
quantization techniques, training paradigm, streamability, and application
domains. We evaluate tokenizers on multiple benchmarks for reconstruction,
downstream performance, and acoustic language modeling, and analyze trade-offs
through controlled ablation studies. Our findings highlight key limitations,
practical considerations, and open challenges, providing insight and guidance
for future research in this rapidly evolving area. For more information,
including our main results and tokenizer database, please refer to our website:
https://poonehmousavi.github.io/dates-website/.
684bd94c3b733ba3336871a2
Pooneh Mousavi
claimed_verified
2025-06-17T07:22:31.425Z
6349a555c85a6b9b0b0563ef
https://cdn-avatars.huggingface.co/v1/production/uploads/1673465674574-6349a555c85a6b9b0b0563ef.png
Pooneh Mousavi
user
poonehmousavi
684bd94c3b733ba3336871a3
Gallil Maimon
claimed_verified
2025-06-13T14:59:26.386Z
66b9bc2dacdbc1d0b39c3b50
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hwR0pVfP_E8XjimXIxDOU.jpeg
Gallil Maimon
user
gallilmaimon
684bd94c3b733ba3336871a4
Adel Moumen
claimed_verified
2025-06-16T09:52:45.561Z
6316f98723ac76623d5b8d0e
https://cdn-avatars.huggingface.co/v1/production/uploads/1674648410294-6316f98723ac76623d5b8d0e.png
Adel Moumen
user
Adel-Moumen
684bd94c3b733ba3336871a5
Darius Petermann
684bd94c3b733ba3336871a6
Jiatong Shi
684bd94c3b733ba3336871a7
Haibin Wu
684bd94c3b733ba3336871a8
Haici Yang
684bd94c3b733ba3336871a9
Anastasia Kuznetsova
684bd94c3b733ba3336871aa
Artem Ploujnikov
684bd94c3b733ba3336871ab
Ricard Marxer
684bd94c3b733ba3336871ac
Bhuvana Ramabhadran
684bd94c3b733ba3336871ad
Benjamin Elizalde
684bd94c3b733ba3336871ae
Loren Lugosch
684bd94c3b733ba3336871af
Jinyu Li
684bd94c3b733ba3336871b0
Cem Subakan
684bd94c3b733ba3336871b1
Phil Woodland
684bd94c3b733ba3336871b2
Minje Kim
684bd94c3b733ba3336871b3
Hung-yi Lee
684bd94c3b733ba3336871b4
Shinji Watanabe
684bd94c3b733ba3336871b5
Yossi Adi
684bd94c3b733ba3336871b6
Mirco Ravanelli
Discrete Audio Tokens: More Than a Survey!
Discrete audio tokens are compact representations that aim to preserve
perceptual quality, phonetic content, and speaker characteristics while
enabling efficient storage and inference, as well as competitive performance
across diverse downstream tasks.They provide a practical alternative to
continuous features, enabling the integration of speech and audio into modern
large language models (LLMs). As interest in token-based audio processing
grows, various tokenization methods have emerged, and several surveys have
reviewed the latest progress in the field. However, existing studies often
focus on specific domains or tasks and lack a unified comparison across various
benchmarks. This paper presents a systematic review and benchmark of discrete
audio tokenizers, covering three domains: speech, music, and general audio. We
propose a taxonomy of tokenization approaches based on encoder-decoder,
quantization techniques, training paradigm, streamability, and application
domains. We evaluate tokenizers on multiple benchmarks for reconstruction,
downstream performance, and acoustic language modeling, and analyze trade-offs
through controlled ablation studies. Our findings highlight key limitations,
practical considerations, and open challenges, providing insight and guidance
for future research in this rapidly evolving area. For more information,
including our main results and tokenizer database, please refer to our website:
https://poonehmousavi.github.io/dates-website/.
A systematic review and benchmark of discrete audio tokenizers across speech, music, and general audio domains is presented, covering their taxonomy, evaluation metrics, and limitations.
discrete audio tokens perceptual quality phonetic content speaker characteristics efficient storage inference large language models tokenization methods encoder-decoder quantization techniques training paradigm streamability application domains reconstruction acoustic language modeling ablation studies