Vision Transformers Don't Need Trained Registers
We investigate the mechanism underlying a previously identified phenomenon in
Vision Transformers -- the emergence of high-norm tokens that lead to noisy
attention maps. We observe that in multiple models (e.g., CLIP, DINOv2), a
sparse set of neurons is responsible for concentrating high-norm activations on
outlier tokens, leading to irregular attention patterns and degrading
downstream visual processing. While the existing solution for removing these
outliers involves retraining models from scratch with additional learned
register tokens, we use our findings to create a training-free approach to
mitigate these artifacts. By shifting the high-norm activations from our
discovered register neurons into an additional untrained token, we can mimic
the effect of register tokens on a model already trained without registers. We
demonstrate that our method produces cleaner attention and feature maps,
enhances performance over base models across multiple downstream visual tasks,
and achieves results comparable to models explicitly trained with register
tokens. We then extend test-time registers to off-the-shelf vision-language
models to improve their interpretability. Our results suggest that test-time
registers effectively take on the role of register tokens at test-time,
offering a training-free solution for any pre-trained model released without
them.
6847ad3b3ec10bdd8ab4df06
Nick Jiang
claimed_verified
2025-06-11T20:21:21.250Z
6398d9d168e3392256aaf952
/avatars/062363a9b5ebb26603c543a7fc3ee4ec.svg
Nick
user
nickjiang
6847ad3b3ec10bdd8ab4df07
Amil Dravid
6847ad3b3ec10bdd8ab4df08
Alexei Efros
6847ad3b3ec10bdd8ab4df09
Yossi Gandelsman
Vision Transformers Don't Need Trained Registers
We investigate the mechanism underlying a previously identified phenomenon in
Vision Transformers -- the emergence of high-norm tokens that lead to noisy
attention maps. We observe that in multiple models (e.g., CLIP, DINOv2), a
sparse set of neurons is responsible for concentrating high-norm activations on
outlier tokens, leading to irregular attention patterns and degrading
downstream visual processing. While the existing solution for removing these
outliers involves retraining models from scratch with additional learned
register tokens, we use our findings to create a training-free approach to
mitigate these artifacts. By shifting the high-norm activations from our
discovered register neurons into an additional untrained token, we can mimic
the effect of register tokens on a model already trained without registers. We
demonstrate that our method produces cleaner attention and feature maps,
enhances performance over base models across multiple downstream visual tasks,
and achieves results comparable to models explicitly trained with register
tokens. We then extend test-time registers to off-the-shelf vision-language
models to improve their interpretability. Our results suggest that test-time
registers effectively take on the role of register tokens at test-time,
offering a training-free solution for any pre-trained model released without
them.
A training-free method shifts high-norm activations in Vision Transformers to an untrained token, enhancing attention maps and performance across visual tasks, and improving interpretability in vision-language models.
Vision Transformers high-norm tokens noisy attention maps activations neurons irregular attention patterns downstream visual processing register tokens feature maps vision-language models interpretability test-time registers