Quantization for OpenAI's Whisper Models: A Comparative Analysis
Automated speech recognition (ASR) models have gained prominence for
applications such as captioning, speech translation, and live transcription.
This paper studies Whisper and two model variants: one optimized for live
speech streaming and another for offline transcription. Notably, these models
have been found to generate hallucinated content, reducing transcription
reliability. Furthermore, larger model variants exhibit increased latency and
pose challenges for deployment on resource-constrained devices. This study
analyzes the similarities and differences between three Whisper models,
qualitatively examining their distinct capabilities. Next, this study
quantifies the impact of model quantization on latency and evaluates its
viability for edge deployment. Using the open source LibriSpeech dataset, this
paper evaluates the word error rate (WER) along with latency analysis of
whispercpp using 3 quantization methods (INT4, INT5, INT8). Results show that
quantization reduces latency by 19\% and model size by 45\%, while preserving
transcription accuracy. These findings provide insights into the optimal use
cases of different Whisper models and edge device deployment possibilities. All
code, datasets, and implementation details are available in a public GitHub
repository: https://github.com/allisonandreyev/WhisperQuantization.git
67d393660d51cf27930a5e5d
Allison Andreyev
extracted_confirmed
2025-03-14T02:25:50.445Z
67d3930e4d3a41ed9f7ac71e
/avatars/4829b1d58376f7987a8891d41108c9c9.svg
Allison Andreyev
user
allisonandreyev
Quantization for OpenAI's Whisper Models: A Comparative Analysis
Automated speech recognition (ASR) models have gained prominence for
applications such as captioning, speech translation, and live transcription.
This paper studies Whisper and two model variants: one optimized for live
speech streaming and another for offline transcription. Notably, these models
have been found to generate hallucinated content, reducing transcription
reliability. Furthermore, larger model variants exhibit increased latency and
pose challenges for deployment on resource-constrained devices. This study
analyzes the similarities and differences between three Whisper models,
qualitatively examining their distinct capabilities. Next, this study
quantifies the impact of model quantization on latency and evaluates its
viability for edge deployment. Using the open source LibriSpeech dataset, this
paper evaluates the word error rate (WER) along with latency analysis of
whispercpp using 3 quantization methods (INT4, INT5, INT8). Results show that
quantization reduces latency by 19\% and model size by 45\%, while preserving
transcription accuracy. These findings provide insights into the optimal use
cases of different Whisper models and edge device deployment possibilities. All
code, datasets, and implementation details are available in a public GitHub
repository: https://github.com/allisonandreyev/WhisperQuantization.git
Whisper models optimized for live streaming and offline transcription are analyzed for their capabilities, latency, and the impact of quantization on edge deployment.
Whisper live speech streaming offline transcription hallucinated content model quantization latency LibriSpeech word error rate WER INT4 INT5 INT8 edge deployment