Masked Audio Text Encoders are Effective Multi-Modal Rescorers
Masked Language Models (MLMs) have proven to be effective for second-pass
rescoring in Automatic Speech Recognition (ASR) systems. In this work, we
propose Masked Audio Text Encoder (MATE), a multi-modal masked language model
rescorer which incorporates acoustic representations into the input space of
MLM. We adopt contrastive learning for effectively aligning the modalities by
learning shared representations. We show that using a multi-modal rescorer is
beneficial for domain generalization of the ASR system when target domain data
is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and
3%-7% on out-of-domain datasets, over the text-only baseline. Additionally,
with very limited amount of training data (0.8 hours), MATE achieves a WER
reduction of 8%-23% over the first-pass baseline.
6462e5ec2538819c72a09a12
Jinglun Cai
claimed_verified
2023-05-17T08:04:57.882Z
6463c26aa5af935cfe70f08d
/avatars/33b1210098891db54f57d1344b5110fb.svg
Jinglun (Jason) Cai
user
jasoncai
6462e5ec2538819c72a09a13
Monica Sunkara
6462e5ec2538819c72a09a14
Xilai Li
6462e5ec2538819c72a09a15
Anshu Bhatia
6462e5ec2538819c72a09a16
Xiao Pan
6462e5ec2538819c72a09a17
Sravan Bodapati
admin_assigned
2023-05-26T09:36:56.541Z
640f6eb0a92fedb0e851c4ed
/avatars/2d2e35ff7d064445ca68a79d71281fd1.svg
Sravan Bodapati
user
sravanbabu1991
Masked Audio Text Encoders are Effective Multi-Modal Rescorers
Masked Language Models (MLMs) have proven to be effective for second-pass
rescoring in Automatic Speech Recognition (ASR) systems. In this work, we
propose Masked Audio Text Encoder (MATE), a multi-modal masked language model
rescorer which incorporates acoustic representations into the input space of
MLM. We adopt contrastive learning for effectively aligning the modalities by
learning shared representations. We show that using a multi-modal rescorer is
beneficial for domain generalization of the ASR system when target domain data
is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and
3%-7% on out-of-domain datasets, over the text-only baseline. Additionally,
with very limited amount of training data (0.8 hours), MATE achieves a WER
reduction of 8%-23% over the first-pass baseline.
A multi-modal masked language model rescorer named MATE, incorporating acoustic representations and using contrastive learning, improves ASR system performance with domain generalization and reduced word error rates.
Masked Language Models multi-modal masked language model acoustic representations contrastive learning WER domain generalization