Image Reconstruction as a Tool for Feature Analysis
Vision encoders are increasingly used in modern applications, from
vision-only models to multimodal systems such as vision-language models.
Despite their remarkable success, it remains unclear how these architectures
represent features internally. Here, we propose a novel approach for
interpreting vision features via image reconstruction. We compare two related
model families, SigLIP and SigLIP2, which differ only in their training
objective, and show that encoders pre-trained on image-based tasks retain
significantly more image information than those trained on non-image tasks such
as contrastive learning. We further apply our method to a range of vision
encoders, ranking them by the informativeness of their feature representations.
Finally, we demonstrate that manipulating the feature space yields predictable
changes in reconstructed images, revealing that orthogonal rotations (rather
than spatial transformations) control color encoding. Our approach can be
applied to any vision encoder, shedding light on the inner structure of its
feature space. The code and model weights to reproduce the experiments are
available in GitHub.
6847d4103ec10bdd8ab4dfb1
Eduard Allakhverdov
claimed_verified
2025-06-10T08:42:38.549Z
6437d0a951c7ebfc813c735b
/avatars/6cbac4e4be5029655702c5d8b9046b90.svg
Allakhverdov Eduard
user
combat-helicopter
6847d4103ec10bdd8ab4dfb2
Dmitrii Tarasov
6847d4103ec10bdd8ab4dfb3
Elizaveta Goncharova
6847d4103ec10bdd8ab4dfb4
Andrey Kuznetsov
claimed_verified
2025-06-12T12:43:01.714Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
Image Reconstruction as a Tool for Feature Analysis
Vision encoders are increasingly used in modern applications, from
vision-only models to multimodal systems such as vision-language models.
Despite their remarkable success, it remains unclear how these architectures
represent features internally. Here, we propose a novel approach for
interpreting vision features via image reconstruction. We compare two related
model families, SigLIP and SigLIP2, which differ only in their training
objective, and show that encoders pre-trained on image-based tasks retain
significantly more image information than those trained on non-image tasks such
as contrastive learning. We further apply our method to a range of vision
encoders, ranking them by the informativeness of their feature representations.
Finally, we demonstrate that manipulating the feature space yields predictable
changes in reconstructed images, revealing that orthogonal rotations (rather
than spatial transformations) control color encoding. Our approach can be
applied to any vision encoder, shedding light on the inner structure of its
feature space. The code and model weights to reproduce the experiments are
available in GitHub.
Image reconstruction reveals that vision encoders retain more image information after image-based tasks and that orthogonal rotations in feature space control color encoding.
SigLIP SigLIP2 vision encoders image reconstruction contrastive learning feature representations