WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained
  Speech Recognition Transformers
Pretrained automatic speech recognition (ASR) models such as Whisper perform
well but still need domain adaptation to handle unseen vocabulary and parlance.
In many real-world settings, collecting speech data is impractical,
necessitating text-only adaptation. We propose WhisTLE, a deeply supervised,
text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE
trains a variational autoencoder (VAE) to model encoder outputs from text and
fine-tunes the decoder using the learned text-to-latent encoder, optionally
combined with text-to-speech (TTS) adaptation. At inference, the original
encoder is restored, incurring no extra runtime cost. Across four out-of-domain
datasets and four ASR models, WhisTLE with TTS reduces word error rate (WER) by
12.3% relative to TTS-only adaptation and outperforms all non-WhisTLE baselines
in 27 of 32 scenarios.
68d0b27a8adc5cd018d15ad6
Akshat Pandey
68d0b27a8adc5cd018d15ad7
Karun Kumar
68d0b27a8adc5cd018d15ad8
Raphael Tang
claimed_verified
2025-09-22T02:29:20.693Z
63250bb8d206fe7b2d2f1b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/1668053024087-63250bb8d206fe7b2d2f1b8b.jpeg
Raphael Tang
user
tetrisd
WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained
  Speech Recognition Transformers
Pretrained automatic speech recognition (ASR) models such as Whisper perform
well but still need domain adaptation to handle unseen vocabulary and parlance.
In many real-world settings, collecting speech data is impractical,
necessitating text-only adaptation. We propose WhisTLE, a deeply supervised,
text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE
trains a variational autoencoder (VAE) to model encoder outputs from text and
fine-tunes the decoder using the learned text-to-latent encoder, optionally
combined with text-to-speech (TTS) adaptation. At inference, the original
encoder is restored, incurring no extra runtime cost. Across four out-of-domain
datasets and four ASR models, WhisTLE with TTS reduces word error rate (WER) by
12.3% relative to TTS-only adaptation and outperforms all non-WhisTLE baselines
in 27 of 32 scenarios.
WhisTLE, a text-only adaptation method using a variational autoencoder, enhances pretrained ASR models with text-to-latent encoding and optional TTS adaptation, reducing word error rates across multiple datasets.
automatic speech recognition ASR Whisper WhisTLE variational autoencoder VAE encoder-decoder text-only adaptation text-to-latent text-to-speech TTS word error rate WER