Tell me why: Visual foundation models as self-explainable classifiers
Visual foundation models (VFMs) have become increasingly popular due to their
state-of-the-art performance. However, interpretability remains crucial for
critical applications. In this sense, self-explainable models (SEM) aim to
provide interpretable classifiers that decompose predictions into a weighted
sum of interpretable concepts. Despite their promise, recent studies have shown
that these explanations often lack faithfulness. In this work, we combine VFMs
with a novel prototypical architecture and specialized training objectives. By
training only a lightweight head (approximately 1M parameters) on top of frozen
VFMs, our approach (ProtoFM) offers an efficient and interpretable solution.
Evaluations demonstrate that our approach achieves competitive classification
performance while outperforming existing models across a range of
interpretability metrics derived from the literature. Code is available at
https://github.com/hturbe/proto-fm.
67c42356054ae6d1c760b643
Hugues Turbé
claimed_verified
2025-03-02T20:15:04.391Z
66588b6fd22637bfab498709
/avatars/9007f0d3b078bd6193912a5359107f24.svg
Hugues Turbé
user
hturbe
67c42356054ae6d1c760b644
Mina Bjelogrlic
67c42356054ae6d1c760b645
Gianmarco Mengaldo
67c42356054ae6d1c760b646
Christian Lovis
Tell me why: Visual foundation models as self-explainable classifiers
Visual foundation models (VFMs) have become increasingly popular due to their
state-of-the-art performance. However, interpretability remains crucial for
critical applications. In this sense, self-explainable models (SEM) aim to
provide interpretable classifiers that decompose predictions into a weighted
sum of interpretable concepts. Despite their promise, recent studies have shown
that these explanations often lack faithfulness. In this work, we combine VFMs
with a novel prototypical architecture and specialized training objectives. By
training only a lightweight head (approximately 1M parameters) on top of frozen
VFMs, our approach (ProtoFM) offers an efficient and interpretable solution.
Evaluations demonstrate that our approach achieves competitive classification
performance while outperforming existing models across a range of
interpretability metrics derived from the literature. Code is available at
https://github.com/hturbe/proto-fm.
ProtoFM combines visual foundation models with a prototypical architecture to achieve competitive classification performance and enhanced interpretability.
Visual foundation models self-explainable models prototypical architecture training objectives interpretable classifiers interpretable concepts interpretability metrics