Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting
  Rare Concepts in Foundation Models
Understanding and mitigating the potential risks associated with foundation
models (FMs) hinges on developing effective interpretability methods. Sparse
Autoencoders (SAEs) have emerged as a promising tool for disentangling FM
representations, but they struggle to capture rare, yet crucial concepts in the
data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to
illuminate these elusive dark matter features by focusing on specific
subdomains. We present a practical recipe for training SSAEs, demonstrating the
efficacy of dense retrieval for data selection and the benefits of Tilted
Empirical Risk Minimization as a training objective to improve concept recall.
Our evaluation of SSAEs on standard metrics, such as downstream perplexity and
L_0 sparsity, show that they effectively capture subdomain tail concepts,
exceeding the capabilities of general-purpose SAEs. We showcase the practical
utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs
achieve a 12.5\% increase in worst-group classification accuracy when applied
to remove spurious gender information. SSAEs provide a powerful new lens for
peering into the inner workings of FMs in subdomains.
67298c6932c59bb502696f25
Aashiq Muhamed
admin_assigned
2024-11-05T16:31:59.109Z
64755a83e0b188d3cb2579d8
/avatars/2c50590905f4bd398a4c9991e1b4b5bb.svg
Aashiq Muhamed
user
aashiqmuhamed
67298c6932c59bb502696f26
Mona Diab
67298c6932c59bb502696f27
Virginia Smith
Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting
  Rare Concepts in Foundation Models
Understanding and mitigating the potential risks associated with foundation
models (FMs) hinges on developing effective interpretability methods. Sparse
Autoencoders (SAEs) have emerged as a promising tool for disentangling FM
representations, but they struggle to capture rare, yet crucial concepts in the
data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to
illuminate these elusive dark matter features by focusing on specific
subdomains. We present a practical recipe for training SSAEs, demonstrating the
efficacy of dense retrieval for data selection and the benefits of Tilted
Empirical Risk Minimization as a training objective to improve concept recall.
Our evaluation of SSAEs on standard metrics, such as downstream perplexity and
L_0 sparsity, show that they effectively capture subdomain tail concepts,
exceeding the capabilities of general-purpose SAEs. We showcase the practical
utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs
achieve a 12.5\% increase in worst-group classification accuracy when applied
to remove spurious gender information. SSAEs provide a powerful new lens for
peering into the inner workings of FMs in subdomains.
Specialized Sparse Autoencoders (SSAEs) effectively capture rare concepts in foundation model representations, improving interpretability and classifier performance on specific subdomains.
Sparse Autoencoders Specialized Sparse Autoencoders disentangling FM representations dense retrieval Tilted Empirical Risk Minimization downstream perplexity $L_0$ sparsity worst-group classification accuracy Bias in Bios dataset