Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo
  Labelling
As the size of pre-trained speech recognition models increases, running these
large models in low-latency or resource-constrained environments becomes
challenging. In this work, we leverage pseudo-labelling to assemble a
large-scale open-source dataset which we use to distill the Whisper model into
a smaller variant, called Distil-Whisper. Using a simple word error rate (WER)
heuristic, we select only the highest quality pseudo-labels for training. The
distilled model is 5.8 times faster with 51% fewer parameters, while performing
to within 1% WER on out-of-distribution test data in a zero-shot transfer
setting. Distil-Whisper maintains the robustness of the Whisper model to
difficult acoustic conditions, while being less prone to hallucination errors
on long-form audio. Distil-Whisper is designed to be paired with Whisper for
speculative decoding, yielding a 2 times speed-up while mathematically ensuring
the same outputs as the original model. To facilitate further research in this
domain, we make our training code, inference code and models publicly
accessible.
65430c759ce0fa3dfa3c1ec6
Sanchit Gandhi
admin_assigned
2023-11-02T10:08:16.773Z
61f91cf54a8e5a275b2b3e7c
https://cdn-avatars.huggingface.co/v1/production/uploads/1653243468328-61f91cf54a8e5a275b2b3e7c.jpeg
Sanchit Gandhi
user
sanchit-gandhi
65430c759ce0fa3dfa3c1ec7
Patrick von Platen
admin_assigned
2023-11-02T10:07:46.528Z
5dfcb1aada6d0311fd3d5448
https://cdn-avatars.huggingface.co/v1/production/uploads/1584435275418-5dfcb1aada6d0311fd3d5448.jpeg
Patrick von Platen
user
patrickvonplaten
65430c759ce0fa3dfa3c1ec8
Alexander M. Rush
admin_assigned
2023-11-02T10:08:05.413Z
5e694f697127cb58dde42fe1
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
Sasha Rush
user
srush
Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo
  Labelling
As the size of pre-trained speech recognition models increases, running these
large models in low-latency or resource-constrained environments becomes
challenging. In this work, we leverage pseudo-labelling to assemble a
large-scale open-source dataset which we use to distill the Whisper model into
a smaller variant, called Distil-Whisper. Using a simple word error rate (WER)
heuristic, we select only the highest quality pseudo-labels for training. The
distilled model is 5.8 times faster with 51% fewer parameters, while performing
to within 1% WER on out-of-distribution test data in a zero-shot transfer
setting. Distil-Whisper maintains the robustness of the Whisper model to
difficult acoustic conditions, while being less prone to hallucination errors
on long-form audio. Distil-Whisper is designed to be paired with Whisper for
speculative decoding, yielding a 2 times speed-up while mathematically ensuring
the same outputs as the original model. To facilitate further research in this
domain, we make our training code, inference code and models publicly
accessible.
Distil-Whisper, a smaller and faster variant of the Whisper model, achieves nearly the same performance with fewer resources and is optimized for low-latency environments.
pseudo-labelling distillation Whisper model word error rate (WER) zero-shot transfer speculative decoding