Representing Speech Through Autoregressive Prediction of Cochlear Tokens
We introduce AuriStream, a biologically inspired model for encoding speech
via a two-stage framework inspired by the human auditory processing hierarchy.
The first stage transforms raw audio into a time-frequency representation based
on the human cochlea, from which we extract discrete cochlear tokens.
The second stage applies an autoregressive sequence model over the cochlear
tokens. AuriStream learns meaningful phoneme and word representations, and
state-of-the-art lexical semantics. AuriStream shows competitive performance on
diverse downstream SUPERB speech tasks. Complementing AuriStream's strong
representational capabilities, it generates continuations of audio which can be
visualized in a spectrogram space and decoded back into audio, providing
insights into the model's predictions. In summary, we present a two-stage
framework for speech representation learning to advance the development of more
human-like models that efficiently handle a range of speech-based tasks.
68a2f5eda4caabb4320e651f
Greta Tuckute
claimed_verified
2025-08-19T13:23:51.680Z
605a5f796ce6cabbb3474b55
/avatars/971866aa240126e480bbe01eff64c009.svg
Greta Tuckute
user
gretatuckute
68a2f5eda4caabb4320e6520
Klemen Kotar
claimed_verified
2025-08-19T08:28:40.445Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
68a2f5eda4caabb4320e6521
Evelina Fedorenko
68a2f5eda4caabb4320e6522
Daniel L. K. Yamins
Representing Speech Through Autoregressive Prediction of Cochlear Tokens
We introduce AuriStream, a biologically inspired model for encoding speech
via a two-stage framework inspired by the human auditory processing hierarchy.
The first stage transforms raw audio into a time-frequency representation based
on the human cochlea, from which we extract discrete cochlear tokens.
The second stage applies an autoregressive sequence model over the cochlear
tokens. AuriStream learns meaningful phoneme and word representations, and
state-of-the-art lexical semantics. AuriStream shows competitive performance on
diverse downstream SUPERB speech tasks. Complementing AuriStream's strong
representational capabilities, it generates continuations of audio which can be
visualized in a spectrogram space and decoded back into audio, providing
insights into the model's predictions. In summary, we present a two-stage
framework for speech representation learning to advance the development of more
human-like models that efficiently handle a range of speech-based tasks.
AuriStream, a biologically inspired two-stage model, encodes speech using cochlear tokens and an autoregressive sequence model, achieving state-of-the-art performance on speech tasks and generating interpretable audio continuations.
cochlear tokens autoregressive sequence model human auditory processing hierarchy human cochlea lexical semantics spectrogram space