FaithfulSAE: Towards Capturing Faithful Features with Sparse
  Autoencoders without External Dataset Dependencies
Sparse Autoencoders (SAEs) have emerged as a promising solution for
decomposing large language model representations into interpretable features.
However, Paulo and Belrose (2025) have highlighted instability across different
initialization seeds, and Heap et al. (2025) have pointed out that SAEs may not
capture model-internal features. These problems likely stem from training SAEs
on external datasets - either collected from the Web or generated by another
model - which may contain out-of-distribution (OOD) data beyond the model's
generalisation capabilities. This can result in hallucinated SAE features,
which we term "Fake Features", that misrepresent the model's internal
activations. To address these issues, we propose FaithfulSAE, a method that
trains SAEs on the model's own synthetic dataset. Using FaithfulSAEs, we
demonstrate that training SAEs on less-OOD instruction datasets results in SAEs
being more stable across seeds. Notably, FaithfulSAEs outperform SAEs trained
on web-based datasets in the SAE probing task and exhibit a lower Fake Feature
Ratio in 5 out of 7 models. Overall, our approach eliminates the dependency on
external datasets, advancing interpretability by better capturing
model-internal features while highlighting the often neglected importance of
SAE training datasets.
685a5c8e0e4ad7e2197586c7
Seonglae Cho
claimed_verified
2025-06-24T08:09:45.757Z
64105805928400b416439f10
https://cdn-avatars.huggingface.co/v1/production/uploads/64105805928400b416439f10/i0jFLo47RTDeNl26hiR9y.jpeg
Seonglae Cho
user
seonglae
685a5c8e0e4ad7e2197586c8
Harryn Oh
685a5c8e0e4ad7e2197586c9
Donghyun Lee
685a5c8e0e4ad7e2197586ca
Luis Eduardo Rodrigues Vieira
685a5c8e0e4ad7e2197586cb
Andrew Bermingham
685a5c8e0e4ad7e2197586cc
Ziad El Sayed
FaithfulSAE: Towards Capturing Faithful Features with Sparse
  Autoencoders without External Dataset Dependencies
Sparse Autoencoders (SAEs) have emerged as a promising solution for
decomposing large language model representations into interpretable features.
However, Paulo and Belrose (2025) have highlighted instability across different
initialization seeds, and Heap et al. (2025) have pointed out that SAEs may not
capture model-internal features. These problems likely stem from training SAEs
on external datasets - either collected from the Web or generated by another
model - which may contain out-of-distribution (OOD) data beyond the model's
generalisation capabilities. This can result in hallucinated SAE features,
which we term "Fake Features", that misrepresent the model's internal
activations. To address these issues, we propose FaithfulSAE, a method that
trains SAEs on the model's own synthetic dataset. Using FaithfulSAEs, we
demonstrate that training SAEs on less-OOD instruction datasets results in SAEs
being more stable across seeds. Notably, FaithfulSAEs outperform SAEs trained
on web-based datasets in the SAE probing task and exhibit a lower Fake Feature
Ratio in 5 out of 7 models. Overall, our approach eliminates the dependency on
external datasets, advancing interpretability by better capturing
model-internal features while highlighting the often neglected importance of
SAE training datasets.
FaithfulSAE improves Sparse Autoencoder stability and interpretability by training on synthetic datasets generated by the model itself, reducing the occurrence of fake features and out-of-distribution data issues.
Sparse Autoencoders SAEs interpretability instability initialization seeds model-internal features out-of-distribution OOD Fake Features SAE probing task synthetic dataset