Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong
  General Audio Event Taggers
In this paper, we focus on Whisper, a recent automatic speech recognition
model trained with a massive 680k hour labeled speech corpus recorded in
diverse conditions. We first show an interesting finding that while Whisper is
very robust against real-world background sounds (e.g., music), its audio
representation is actually not noise-invariant, but is instead highly
correlated to non-speech sounds, indicating that Whisper recognizes speech
conditioned on the noise type. With this finding, we build a unified audio
tagging and speech recognition model Whisper-AT by freezing the backbone of
Whisper, and training a lightweight audio tagging model on top of it. With <1%
extra computational cost, Whisper-AT can recognize audio events, in addition to
spoken text, in a single forward pass.
64a77fb21fc2c199e62f7b97
Yuan Gong
claimed_verified
2023-07-10T09:24:30.286Z
63b5db8ebda8d44adf35285e
/avatars/24c98c05509fa7b1ac3371ceb1b81f67.svg
Yuan Gong
user
yuangongfdu
64a77fb21fc2c199e62f7b98
Sameer Khurana
admin_assigned
2023-07-07T13:16:54.592Z
62050bb6a03b2e13bf5401cc
/avatars/d45580d0a858e6194dc169169b0a2d57.svg
Sameer Khurana
user
skhurana
64a77fb21fc2c199e62f7b99
Leonid Karlinsky
64a77fb21fc2c199e62f7b9a
James Glass
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong
  General Audio Event Taggers
In this paper, we focus on Whisper, a recent automatic speech recognition
model trained with a massive 680k hour labeled speech corpus recorded in
diverse conditions. We first show an interesting finding that while Whisper is
very robust against real-world background sounds (e.g., music), its audio
representation is actually not noise-invariant, but is instead highly
correlated to non-speech sounds, indicating that Whisper recognizes speech
conditioned on the noise type. With this finding, we build a unified audio
tagging and speech recognition model Whisper-AT by freezing the backbone of
Whisper, and training a lightweight audio tagging model on top of it. With <1%
extra computational cost, Whisper-AT can recognize audio events, in addition to
spoken text, in a single forward pass.
Whisper-AT, a unified model built upon Whisper, extends speech recognition to audio event tagging with minimal extra computational cost.
automatic speech recognition Whisper labeled speech corpus background sounds audio representation noise-invariant non-speech sounds audio tagging backbone forward pass