Attention Sinks in Diffusion Language Models
Masked Diffusion Language Models (DLMs) have recently emerged as a promising
alternative to traditional Autoregressive Models (ARMs). DLMs employ
transformer encoders with bidirectional attention, enabling parallel token
generation while maintaining competitive performance. Although their efficiency
and effectiveness have been extensively studied, the internal mechanisms that
govern DLMs remain largely unexplored. In this work, we conduct an empirical
analysis of DLM attention patterns, focusing on the attention sinking
phenomenon, an effect previously observed in various transformer-based
architectures. Our findings reveal that DLMs also exhibit attention sinks, but
with distinct characteristics. First, unlike in ARMs, the sink positions in
DLMs tend to shift throughout the generation process, displaying a dynamic
behaviour. Second, while ARMs are highly sensitive to the removal of attention
sinks, DLMs remain robust: masking sinks leads to only a minor degradation in
performance. These results provide new insights into the inner workings of
diffusion-based language models and highlight fundamental differences in how
they allocate and utilize attention compared to autoregressive models.
68f89d937669bcaeecce0f2f
Maximo Eduardo Rulli
claimed_verified
2025-10-22T15:42:15.171Z
643986457245b3b7f453aec7
/avatars/37b3fb9a67de7674c898788999a12265.svg
Maximo Rulli
user
maximorulli
68f89d937669bcaeecce0f30
Simone Petruzzi
68f89d937669bcaeecce0f31
Edoardo Michielon
68f89d937669bcaeecce0f32
Fabrizio Silvestri
68f89d937669bcaeecce0f33
Simone Scardapane
68f89d937669bcaeecce0f34
Alessio Devoto
Attention Sinks in Diffusion Language Models
Masked Diffusion Language Models (DLMs) have recently emerged as a promising
alternative to traditional Autoregressive Models (ARMs). DLMs employ
transformer encoders with bidirectional attention, enabling parallel token
generation while maintaining competitive performance. Although their efficiency
and effectiveness have been extensively studied, the internal mechanisms that
govern DLMs remain largely unexplored. In this work, we conduct an empirical
analysis of DLM attention patterns, focusing on the attention sinking
phenomenon, an effect previously observed in various transformer-based
architectures. Our findings reveal that DLMs also exhibit attention sinks, but
with distinct characteristics. First, unlike in ARMs, the sink positions in
DLMs tend to shift throughout the generation process, displaying a dynamic
behaviour. Second, while ARMs are highly sensitive to the removal of attention
sinks, DLMs remain robust: masking sinks leads to only a minor degradation in
performance. These results provide new insights into the inner workings of
diffusion-based language models and highlight fundamental differences in how
they allocate and utilize attention compared to autoregressive models.
Empirical analysis of Masked Diffusion Language Models (DLMs) reveals distinct attention sinking phenomena and robustness compared to Autoregressive Models (ARMs).
Masked Diffusion Language Models DLMs Autoregressive Models ARMs transformer encoders bidirectional attention parallel token generation attention sinking attention patterns dynamic behaviour performance degradation