Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Diffusion-based large language models (DLLMs) have recently attracted growing
interest as an alternative to autoregressive decoders. In this work, we present
an empirical study on using the diffusion-based large language model LLaDA for
automatic speech recognition (ASR). We first investigate its use as an external
deliberation-based processing module for Whisper-LLaMA transcripts. By
leveraging the bidirectional attention and denoising capabilities of LLaDA, we
explore random masking, low-confidence masking, and semi-autoregressive
strategies, showing that Whisper-LLaDA substantially reduces WER compared with
the baseline. On LibriSpeech, the best cascade system achieves 2.25%/4.94% WER
on test-clean/test-other, representing a 12.3% relative improvement over the
Whisper-LLaMA baseline on the test-other split. In contrast, a plain-text LLaDA
without acoustic features fails to improve accuracy, highlighting the
importance of audio-conditioned embeddings. We further evaluate Whisper-LLaDA
as a standalone decoder for ASR with diffusion-based and semi-autoregressive
decoding. Most experimental configurations achieve faster inference than the
Whisper-LLaMA baseline, although recognition accuracy is slightly lower. These
findings offer an empirical view of diffusion-based LLMs for ASR and point to
promising directions for improvements.
68d1f4541ca7156988a8ec76
Mengqi Wang
68d1f4541ca7156988a8ec77
Zhan Liu
claimed_verified
2025-09-23T02:40:36.928Z
660bb388aeb2c22d9bd3894a
https://cdn-avatars.huggingface.co/v1/production/uploads/660bb388aeb2c22d9bd3894a/zSmagaNp6yXovj8R8-beT.jpeg
Jason Liu
user
liuzhan22
68d1f4541ca7156988a8ec78
Zengrui Jin
68d1f4541ca7156988a8ec79
Guangzhi Sun
68d1f4541ca7156988a8ec7a
Chao Zhang
68d1f4541ca7156988a8ec7b
Philip C. Woodland
Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Diffusion-based large language models (DLLMs) have recently attracted growing
interest as an alternative to autoregressive decoders. In this work, we present
an empirical study on using the diffusion-based large language model LLaDA for
automatic speech recognition (ASR). We first investigate its use as an external
deliberation-based processing module for Whisper-LLaMA transcripts. By
leveraging the bidirectional attention and denoising capabilities of LLaDA, we
explore random masking, low-confidence masking, and semi-autoregressive
strategies, showing that Whisper-LLaDA substantially reduces WER compared with
the baseline. On LibriSpeech, the best cascade system achieves 2.25%/4.94% WER
on test-clean/test-other, representing a 12.3% relative improvement over the
Whisper-LLaMA baseline on the test-other split. In contrast, a plain-text LLaDA
without acoustic features fails to improve accuracy, highlighting the
importance of audio-conditioned embeddings. We further evaluate Whisper-LLaDA
as a standalone decoder for ASR with diffusion-based and semi-autoregressive
decoding. Most experimental configurations achieve faster inference than the
Whisper-LLaMA baseline, although recognition accuracy is slightly lower. These
findings offer an empirical view of diffusion-based LLMs for ASR and point to
promising directions for improvements.
Diffusion-based large language models improve automatic speech recognition through bidirectional attention and denoising, showing better performance when integrated with Whisper-LLaMA transcripts and audio-conditioned embeddings.
diffusion-based large language models DLLMs Whisper-LLaMA bidirectional attention denoising random masking low-confidence masking semi-autoregressive strategies WER LibriSpeech audio-conditioned embeddings diffusion-based decoding