Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders
Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi
Recent interpretability work on large language models (LLMs) is increasingly dominated by a feature discovery approach with the help of proxy modules, where the quality of features learned by, e.g., sparse auto-encoders (SAEs), has been evaluated.This paradigm naturally raises a critical question --- how much better features such proxies have indeed discovered, especially compared to those already represented within the original model parameters --- and unfortunately, such a comparison has little been made so far.In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks.Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability scores than the SAEs, and features discovered in SAEs and FFs diverged.These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors.