DinoSR: Self-Distillation and Online Clustering for Self-supervised
  Speech Representation Learning
In this paper, we introduce self-distillation and online clustering for
self-supervised speech representation learning (DinoSR) which combines masked
language modeling, self-distillation, and online clustering. We show that these
concepts complement each other and result in a strong representation learning
model for speech. DinoSR first extracts contextualized embeddings from the
input audio with a teacher network, then runs an online clustering system on
the embeddings to yield a machine-discovered phone inventory, and finally uses
the discretized tokens to guide a student network. We show that DinoSR
surpasses previous state-of-the-art performance in several downstream tasks,
and provide a detailed analysis of the model and the learned discrete units.
The source code will be made available after the anonymity period.
646586505cdb9ab50f874fdc
Alexander H. Liu
646586505cdb9ab50f874fdd
Heng-Jui Chang
claimed_verified
2023-05-19T10:28:27.902Z
6179f36a2a4e9edab3a95798
https://cdn-avatars.huggingface.co/v1/production/uploads/6179f36a2a4e9edab3a95798/0mmFY5lFzPC5k6_GSdmYQ.jpeg
Heng-Jui Chang
user
vectominist
646586505cdb9ab50f874fde
Michael Auli
admin_assigned
2023-05-26T09:19:25.810Z
6176e9d475f6ae94d250042c
/avatars/ce0eafa09525b741c22717bbde6c6a16.svg
Michael Auli
user
michaelauli
646586505cdb9ab50f874fdf
Wei-Ning Hsu
646586505cdb9ab50f874fe0
James R. Glass
DinoSR: Self-Distillation and Online Clustering for Self-supervised
  Speech Representation Learning
In this paper, we introduce self-distillation and online clustering for
self-supervised speech representation learning (DinoSR) which combines masked
language modeling, self-distillation, and online clustering. We show that these
concepts complement each other and result in a strong representation learning
model for speech. DinoSR first extracts contextualized embeddings from the
input audio with a teacher network, then runs an online clustering system on
the embeddings to yield a machine-discovered phone inventory, and finally uses
the discretized tokens to guide a student network. We show that DinoSR
surpasses previous state-of-the-art performance in several downstream tasks,
and provide a detailed analysis of the model and the learned discrete units.
The source code will be made available after the anonymity period.
Self-distillation and online clustering enhance self-supervised speech representation learning by combining masked language modeling, contextualized embeddings, and discrete tokens, achieving superior performance in various downstream tasks.
self-distillation online clustering self-supervised speech representation learning masked language modeling contextualized embeddings teacher network student network discrete tokens