Signals in the Stream: Pinpointing Attention-Causal Communication in Language Models
Gabriel Franco, Mark Crovella
The attention mechanism plays a central role in the computations performed by transformer-based models, and understanding the reasons why heads attend to specific tokens can aid in interpretability of language models. Although considerable work has shown that models construct low-dimensional feature representations, little work has explicitly tied low-dimensional features to the attention mechanism itself. In this paper we work to bridge this gap by presenting methods for identifying attention-causal communication, meaning low-dimensional features that are written into and read from tokens, and that have a provable causal relationship to attention patterns. Our methods do not make use of counterfactual inputs to the model, but rather work by joint analysis of model weights and model representations to isolate signals -- attention-causal features.  These signals serve as communication channels from upstream model components to the attention head.  We show that by identifying the signals present when a model processes a prompt, we can perform circuit discovery that is more precise and efficient than previous methods.  Further, we show that signals can uncover unexplored mechanisms at work in the model, including a surprising degree of global coordination across attention heads.