The Structure of Relation Decoding Linear Operators in Large Language Models
Miranda Anna Christ, Adrián Csiszárik, Gergely Becsó, Dániel Varga
This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple rank-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., primary language spoken in a country and urban area in country are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlight why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.