StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient
  SpeechLLMs
Prevalent semantic speech tokenizers, designed to capture linguistic content,
are surprisingly fragile. We find they are not robust to meaning-irrelevant
acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech
is perfectly intelligible, their output token sequences can change drastically,
increasing the learning burden for downstream LLMs. This instability stems from
two flaws: a brittle single-path quantization architecture and a distant
training signal indifferent to intermediate token stability. To address this,
we introduce StableToken, a tokenizer that achieves stability through a
consensus-driven mechanism. Its multi-branch architecture processes audio in
parallel, and these representations are merged via a powerful bit-wise voting
mechanism to form a single, stable token sequence. StableToken sets a new
state-of-the-art in token stability, drastically reducing Unit Edit Distance
(UED) under diverse noise conditions. This foundational stability translates
directly to downstream benefits, significantly improving the robustness of
SpeechLLMs on a variety of tasks.
68dba222d2bf1f4b15ec792e
Yuhan Song
claimed_verified
2025-09-30T10:17:34.115Z
62b076f6bcd06fe282983c36
/avatars/0086306236782ac1ce241b296ad215b2.svg
Yuhan SONG
user
QbethQ
68dba222d2bf1f4b15ec792f
Linhao Zhang
68dba222d2bf1f4b15ec7930
Chuhan Wu
68dba222d2bf1f4b15ec7931
Aiwei Liu
68dba222d2bf1f4b15ec7932
Wei Jia
68dba222d2bf1f4b15ec7933
Houfeng Wang
68dba222d2bf1f4b15ec7934
Xiao Zhou
StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient
  SpeechLLMs
Prevalent semantic speech tokenizers, designed to capture linguistic content,
are surprisingly fragile. We find they are not robust to meaning-irrelevant
acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech
is perfectly intelligible, their output token sequences can change drastically,
increasing the learning burden for downstream LLMs. This instability stems from
two flaws: a brittle single-path quantization architecture and a distant
training signal indifferent to intermediate token stability. To address this,
we introduce StableToken, a tokenizer that achieves stability through a
consensus-driven mechanism. Its multi-branch architecture processes audio in
parallel, and these representations are merged via a powerful bit-wise voting
mechanism to form a single, stable token sequence. StableToken sets a new
state-of-the-art in token stability, drastically reducing Unit Edit Distance
(UED) under diverse noise conditions. This foundational stability translates
directly to downstream benefits, significantly improving the robustness of
SpeechLLMs on a variety of tasks.
StableToken, a multi-branch consensus-driven tokenizer, enhances token stability and robustness in speech processing, improving SpeechLLMs' performance under noisy conditions.
semantic speech tokenizers acoustic perturbations Signal-to-Noise Ratios SNRs learning burden downstream LLMs brittle single-path quantization training signal multi-branch architecture bit-wise voting mechanism token stability Unit Edit Distance UED SpeechLLMs