Revisiting LRP: Positional Attribution as the Missing Ingredient for
  Transformer Explainability
The development of effective explainability tools for Transformers is a
crucial pursuit in deep learning research. One of the most promising approaches
in this domain is Layer-wise Relevance Propagation (LRP), which propagates
relevance scores backward through the network to the input space by
redistributing activation values based on predefined rules. However, existing
LRP-based methods for Transformer explainability entirely overlook a critical
component of the Transformer architecture: its positional encoding (PE),
resulting in violation of the conservation property, and the loss of an
important and unique type of relevance, which is also associated with
structural and positional features. To address this limitation, we reformulate
the input space for Transformer explainability as a set of position-token
pairs. This allows us to propose specialized theoretically-grounded LRP rules
designed to propagate attributions across various positional encoding methods,
including Rotary, Learnable, and Absolute PE. Extensive experiments with both
fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3,
demonstrate that our method significantly outperforms the state-of-the-art in
both vision and NLP explainability tasks. Our code is publicly available.
683fe27c4f32bd7bbca087fc
Yarden Bakish
683fe27c4f32bd7bbca087fd
Itamar Zimerman
admin_assigned
2025-06-05T07:32:51.961Z
65376feed325b3f02fb92c69
/avatars/e952918cf434d5302e9b1a404eccaf0e.svg
Itamar Zimerman
user
ItamarZ
683fe27c4f32bd7bbca087fe
Hila Chefer
683fe27c4f32bd7bbca087ff
Lior Wolf
admin_assigned
2025-06-05T07:33:06.122Z
62f9f170363251ee40a08046
/avatars/67ad459db44b605be0c1f6bd33363d6e.svg
Lior Wolf
user
liorwolf
Revisiting LRP: Positional Attribution as the Missing Ingredient for
  Transformer Explainability
The development of effective explainability tools for Transformers is a
crucial pursuit in deep learning research. One of the most promising approaches
in this domain is Layer-wise Relevance Propagation (LRP), which propagates
relevance scores backward through the network to the input space by
redistributing activation values based on predefined rules. However, existing
LRP-based methods for Transformer explainability entirely overlook a critical
component of the Transformer architecture: its positional encoding (PE),
resulting in violation of the conservation property, and the loss of an
important and unique type of relevance, which is also associated with
structural and positional features. To address this limitation, we reformulate
the input space for Transformer explainability as a set of position-token
pairs. This allows us to propose specialized theoretically-grounded LRP rules
designed to propagate attributions across various positional encoding methods,
including Rotary, Learnable, and Absolute PE. Extensive experiments with both
fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3,
demonstrate that our method significantly outperforms the state-of-the-art in
both vision and NLP explainability tasks. Our code is publicly available.
A specialized LRP method for Transformer explainability considers positional encoding, improving relevance propagation and outperforming existing methods.
Layer-wise Relevance Propagation (LRP) Transformers positional encoding (PE) Rotary Learnable Absolute PE vision NLP explainability tasks