BlockDecoder: Boosting ASR Decoders with Context and Merger Modules
Darshan Prabhu, Preethi Jyothi
Attention-based encoder decoder models are the backbone of state-of-the-art architectures for automatic speech recognition (ASR). These models combine a powerful encoder that extracts rich acoustic features with a decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe a systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the latter layers focus largely on merging acoustic and textual information to make accurate predictions. Leveraging this key insight, we propose **BlockDecoder**, a novel decoder architecture comprising two distinct components: a **TextEncoder** that is purely text-based, and a  **Merger** that autoregressively generates tokens in blockwise fashion by combining information from the audio encoder and **TextEncoder**. These two components of **BlockDecoder** collectively result in substantial latency gains. Across diverse datasets, languages and speech tasks, we demonstrate that our proposed **BlockDecoder** achieves a significant speedup ($\sim 2$x) compared to traditional decoders, without any degradation in performance.