Mechanistic Permutability: Match Features Across Layers
Understanding how features evolve across layers in deep neural networks is a
fundamental challenge in mechanistic interpretability, particularly due to
polysemanticity and feature superposition. While Sparse Autoencoders (SAEs)
have been used to extract interpretable features from individual layers,
aligning these features across layers has remained an open problem. In this
paper, we introduce SAE Match, a novel, data-free method for aligning SAE
features across different layers of a neural network. Our approach involves
matching features by minimizing the mean squared error between the folded
parameters of SAEs, a technique that incorporates activation thresholds into
the encoder and decoder weights to account for differences in feature scales.
Through extensive experiments on the Gemma 2 language model, we demonstrate
that our method effectively captures feature evolution across layers, improving
feature matching quality. We also show that features persist over several
layers and that our approach can approximate hidden states across layers. Our
work advances the understanding of feature dynamics in neural networks and
provides a new tool for mechanistic interpretability studies.
670cd93f9cf5450e15cfe12d
Nikita Balagansky
claimed_verified
2024-10-14T10:47:41.728Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
670cd93f9cf5450e15cfe12e
Ian Maksimov
claimed_verified
2024-10-15T09:11:15.152Z
65cbcb760b3bd8f5ce402bd5
/avatars/2a60e344cf219eed58217ccf5db26680.svg
Ian Maksimov
user
BayesianMonster
670cd93f9cf5450e15cfe12f
Daniil Gavrilov
claimed_verified
2024-10-14T08:43:11.835Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
Mechanistic Permutability: Match Features Across Layers
Understanding how features evolve across layers in deep neural networks is a
fundamental challenge in mechanistic interpretability, particularly due to
polysemanticity and feature superposition. While Sparse Autoencoders (SAEs)
have been used to extract interpretable features from individual layers,
aligning these features across layers has remained an open problem. In this
paper, we introduce SAE Match, a novel, data-free method for aligning SAE
features across different layers of a neural network. Our approach involves
matching features by minimizing the mean squared error between the folded
parameters of SAEs, a technique that incorporates activation thresholds into
the encoder and decoder weights to account for differences in feature scales.
Through extensive experiments on the Gemma 2 language model, we demonstrate
that our method effectively captures feature evolution across layers, improving
feature matching quality. We also show that features persist over several
layers and that our approach can approximate hidden states across layers. Our
work advances the understanding of feature dynamics in neural networks and
provides a new tool for mechanistic interpretability studies.
A data-free method called SAE Match aligns Sparse Autoencoder features across neural network layers, capturing feature evolution and approximation of hidden states.
Sparse Autoencoders SAE Match dense connections feature matching feature evolution feature persistence hidden states Gemma 2 language model