Accelerating Transducers through Adjacent Token Merging
Recent end-to-end automatic speech recognition (ASR) systems often utilize a
Transformer-based acoustic encoder that generates embedding at a high frame
rate. However, this design is inefficient, particularly for long speech signals
due to the quadratic computation of self-attention. To address this, we propose
a new method, Adjacent Token Merging (A-ToMe), which gradually combines
adjacent tokens with high similarity scores between their key values. In this
way, the total time step could be reduced, and the inference of both the
encoder and joint network is accelerated. Experiments on LibriSpeech show that
our method can reduce 57% of tokens and improve the inference speed on GPU by
70% without any notable loss of accuracy. Additionally, we demonstrate that
A-ToMe is also an effective solution to reduce tokens in long-form ASR, where
the input speech consists of multiple utterances.
649cf693bc919c0fd744d2c0
Yuang Li
649cf693bc919c0fd744d2c1
Yu Wu
649cf693bc919c0fd744d2c2
Jinyu Li
649cf693bc919c0fd744d2c3
Shujie Liu
Accelerating Transducers through Adjacent Token Merging
Recent end-to-end automatic speech recognition (ASR) systems often utilize a
Transformer-based acoustic encoder that generates embedding at a high frame
rate. However, this design is inefficient, particularly for long speech signals
due to the quadratic computation of self-attention. To address this, we propose
a new method, Adjacent Token Merging (A-ToMe), which gradually combines
adjacent tokens with high similarity scores between their key values. In this
way, the total time step could be reduced, and the inference of both the
encoder and joint network is accelerated. Experiments on LibriSpeech show that
our method can reduce 57% of tokens and improve the inference speed on GPU by
70% without any notable loss of accuracy. Additionally, we demonstrate that
A-ToMe is also an effective solution to reduce tokens in long-form ASR, where
the input speech consists of multiple utterances.
A-ToMe reduces computational costs in ASR systems by merging adjacent tokens with high similarity, thereby accelerating inference without sacrificing accuracy.
Transformer-based acoustic encoder self-attention adjacent token merging A-ToMe LibriSpeech long-form ASR