Sparse Autoencoders Learn Monosemantic Features in Vision-Language
  Models
Sparse Autoencoders (SAEs) have recently been shown to enhance
interpretability and steerability in Large Language Models (LLMs). In this
work, we extend the application of SAEs to Vision-Language Models (VLMs), such
as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
in vision representations. Our experimental results reveal that SAEs trained on
VLMs significantly enhance the monosemanticity of individual neurons while also
exhibiting hierarchical representations that align well with expert-defined
structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
applying SAEs to intervene on a CLIP vision encoder, directly steer output from
multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
model. These findings emphasize the practicality and efficacy of SAEs as an
unsupervised approach for enhancing both the interpretability and control of
VLMs.
67ef9a8885ea9d1d7db3fdb5
Mateusz Pach
67ef9a8885ea9d1d7db3fdb6
Shyamgopal Karthik
claimed_verified
2025-04-08T06:54:15.309Z
6254599b6e36fe62e141c8f9
/avatars/08d6b68b92c7bfd0a8be022ba9f2f289.svg
Shyamgopal Karthik
user
shyamgopal
67ef9a8885ea9d1d7db3fdb7
Quentin Bouniot
claimed_verified
2025-06-23T08:33:18.968Z
63076524cd148dbc5e4c67ec
/avatars/8c9ef9ee946364dcd3a3d3d5f32cb7e5.svg
Quentin Bouniot
user
qbouniot
67ef9a8885ea9d1d7db3fdb8
Serge Belongie
67ef9a8885ea9d1d7db3fdb9
Zeynep Akata
Sparse Autoencoders Learn Monosemantic Features in Vision-Language
  Models
Sparse Autoencoders (SAEs) have recently been shown to enhance
interpretability and steerability in Large Language Models (LLMs). In this
work, we extend the application of SAEs to Vision-Language Models (VLMs), such
as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
in vision representations. Our experimental results reveal that SAEs trained on
VLMs significantly enhance the monosemanticity of individual neurons while also
exhibiting hierarchical representations that align well with expert-defined
structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
applying SAEs to intervene on a CLIP vision encoder, directly steer output from
multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
model. These findings emphasize the practicality and efficacy of SAEs as an
unsupervised approach for enhancing both the interpretability and control of
VLMs.
Sparse Autoencoders (SAEs) enhance monosemanticity and hierarchical representations in Vision-Language Models (VLMs) such as CLIP, allowing intervention without modifying the underlying model.
Sparse Autoencoders SAEs Vision-Language Models VLMs CLIP multimodal LLMs interpretability monosemanticity hierarchical representations iNaturalist taxonomy LLaVA