OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features
Sparse autoencoders (SAEs) are a technique for sparse decomposition of neural
network activations into human-interpretable features. However, current SAEs
suffer from feature absorption, where specialized features capture instances of
general features creating representation holes, and feature composition, where
independent features merge into composite representations. In this work, we
introduce Orthogonal SAE (OrtSAE), a novel approach aimed to mitigate these
issues by enforcing orthogonality between the learned features. By implementing
a new training procedure that penalizes high pairwise cosine similarity between
SAE features, OrtSAE promotes the development of disentangled features while
scaling linearly with the SAE size, avoiding significant computational
overhead. We train OrtSAE across different models and layers and compare it
with other methods. We find that OrtSAE discovers 9% more distinct features,
reduces feature absorption (by 65%) and composition (by 15%), improves
performance on spurious correlation removal (+6%), and achieves on-par
performance for other downstream tasks compared to traditional SAEs.
68e3951373e20ab57784217a
Anton Korznikov
68e3951373e20ab57784217b
Andrey Galichin
68e3951373e20ab57784217c
Alexey Dontsov
claimed_verified
2025-10-07T12:28:18.429Z
60cd95ee15ecba5f2200304a
https://cdn-avatars.huggingface.co/v1/production/uploads/60cd95ee15ecba5f2200304a/3gMYeWm8wQO5KfqE5RmEe.jpeg
Alexey Dontsov
user
therem
68e3951373e20ab57784217d
Oleg Rogov
68e3951373e20ab57784217e
Elena Tutubalina
claimed_verified
2025-10-18T04:50:55.406Z
662f8d645c4db70c77a203b0
/avatars/72f9a3c39b3ba5114388d16a35524835.svg
Elena Tutubalina
user
tlenusik
68e3951373e20ab57784217f
Ivan Oseledets
OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features
Sparse autoencoders (SAEs) are a technique for sparse decomposition of neural
network activations into human-interpretable features. However, current SAEs
suffer from feature absorption, where specialized features capture instances of
general features creating representation holes, and feature composition, where
independent features merge into composite representations. In this work, we
introduce Orthogonal SAE (OrtSAE), a novel approach aimed to mitigate these
issues by enforcing orthogonality between the learned features. By implementing
a new training procedure that penalizes high pairwise cosine similarity between
SAE features, OrtSAE promotes the development of disentangled features while
scaling linearly with the SAE size, avoiding significant computational
overhead. We train OrtSAE across different models and layers and compare it
with other methods. We find that OrtSAE discovers 9% more distinct features,
reduces feature absorption (by 65%) and composition (by 15%), improves
performance on spurious correlation removal (+6%), and achieves on-par
performance for other downstream tasks compared to traditional SAEs.
Orthogonal Sparse Autoencoders (OrtSAE) mitigate feature absorption and composition by enforcing orthogonality, leading to better feature discovery and improved performance on spurious correlation removal.
sparse autoencoders feature absorption feature composition orthogonal SAE OrtSAE disentangled features cosine similarity spurious correlation removal