LiteASR: Efficient Automatic Speech Recognition with Low-Rank
  Approximation
Modern automatic speech recognition (ASR) models, such as OpenAI's Whisper,
rely on deep encoder-decoder architectures, and their encoders are a critical
bottleneck for efficient deployment due to high computational intensity. We
introduce LiteASR, a low-rank compression scheme for ASR encoders that
significantly reduces inference costs while maintaining transcription accuracy.
Our approach leverages the strong low-rank properties observed in intermediate
activations: by applying principal component analysis (PCA) with a small
calibration dataset, we approximate linear transformations with a chain of
low-rank matrix multiplications, and further optimize self-attention to work in
the reduced dimension. Evaluation results show that our method can compress
Whisper large-v3's encoder size by over 50%, matching Whisper medium's size
with better transcription accuracy, thereby establishing a new Pareto-optimal
frontier of efficiency and performance. The code of LiteASR is available at
https://github.com/efeslab/LiteASR.
67c516998d02783fa3a52dc8
Keisuke Kamahori
claimed_verified
2025-03-03T08:07:02.986Z
6304ac1a412a1b9d381ca378
/avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
Keisuke Kamahori
user
kamahori
67c516998d02783fa3a52dc9
Jungo Kasai
admin_assigned
2025-03-03T09:43:49.097Z
62908273c740ebb981a6dba4
/avatars/465f50369c367b07670f5209c83d65f2.svg
Jungo Kasai
user
jungok
67c516998d02783fa3a52dca
Noriyuki Kojima
admin_assigned
2025-03-03T09:43:56.698Z
628c26a8b80bb09700d6af86
https://cdn-avatars.huggingface.co/v1/production/uploads/1653352051245-noauth.jpeg
Noriyuki Kojima
user
kojimano
67c516998d02783fa3a52dcb
Baris Kasikci
admin_assigned
2025-03-03T09:44:04.084Z
654132fe5a9a913c6c870e79
/avatars/2f6807eddef1929c571977e9af35f952.svg
Baris Kasikci
user
kasikci
LiteASR: Efficient Automatic Speech Recognition with Low-Rank
  Approximation
Modern automatic speech recognition (ASR) models, such as OpenAI's Whisper,
rely on deep encoder-decoder architectures, and their encoders are a critical
bottleneck for efficient deployment due to high computational intensity. We
introduce LiteASR, a low-rank compression scheme for ASR encoders that
significantly reduces inference costs while maintaining transcription accuracy.
Our approach leverages the strong low-rank properties observed in intermediate
activations: by applying principal component analysis (PCA) with a small
calibration dataset, we approximate linear transformations with a chain of
low-rank matrix multiplications, and further optimize self-attention to work in
the reduced dimension. Evaluation results show that our method can compress
Whisper large-v3's encoder size by over 50%, matching Whisper medium's size
with better transcription accuracy, thereby establishing a new Pareto-optimal
frontier of efficiency and performance. The code of LiteASR is available at
https://github.com/efeslab/LiteASR.
LiteASR reduces ASR encoder size and inference costs while improving accuracy through low-rank compression and optimized self-attention.
encoder-decoder architectures low-rank compression principal component analysis PCA self-attention Pareto-optimal frontier