Denoising Vision Transformers
We delve into a nuanced but significant challenge inherent to Vision
Transformers (ViTs): feature maps of these models exhibit grid-like artifacts,
which detrimentally hurt the performance of ViTs in downstream tasks. Our
investigations trace this fundamental issue down to the positional embeddings
at the input stage. To address this, we propose a novel noise model, which is
universally applicable to all ViTs. Specifically, the noise model dissects ViT
outputs into three components: a semantics term free from noise artifacts and
two artifact-related terms that are conditioned on pixel locations. Such a
decomposition is achieved by enforcing cross-view feature consistency with
neural fields in a per-image basis. This per-image optimization process
extracts artifact-free features from raw ViT outputs, providing clean features
for offline applications. Expanding the scope of our solution to support online
functionality, we introduce a learnable denoiser to predict artifact-free
features directly from unprocessed ViT outputs, which shows remarkable
generalization capabilities to novel data without the need for per-image
optimization. Our two-stage approach, termed Denoising Vision Transformers
(DVT), does not require re-training existing pre-trained ViTs and is
immediately applicable to any Transformer-based architecture. We evaluate our
method on a variety of representative ViTs (DINO, MAE, DeiT-III, EVA02, CLIP,
DINOv2, DINOv2-reg). Extensive evaluations demonstrate that our DVT
consistently and significantly improves existing state-of-the-art
general-purpose models in semantic and geometric tasks across multiple datasets
(e.g., +3.84 mIoU). We hope our study will encourage a re-evaluation of ViT
design, especially regarding the naive use of positional embeddings.
659b66cba7f2d24917e1788a
Jiawei Yang
admin_assigned
2024-01-08T11:39:48.545Z
6594e1079e16fa75101bda67
/avatars/8cffecfeaef6dcec92c20fcdfbb89e8a.svg
Jiawei Yang
user
jjiaweiyang
659b66cba7f2d24917e1788b
Katie Z Luo
659b66cba7f2d24917e1788c
Jiefeng Li
admin_assigned
2024-01-08T11:41:20.137Z
630b5be6a20a5367811a0a2f
https://cdn-avatars.huggingface.co/v1/production/uploads/1664521386819-630b5be6a20a5367811a0a2f.jpeg
Jiefeng Li
user
JeffLi
659b66cba7f2d24917e1788d
Kilian Q Weinberger
admin_assigned
2024-01-08T11:41:33.983Z
60a7f4b44e24361791fe8238
/avatars/c33475f7740d6890733654b002a2d2ad.svg
Weinberger
user
Kilian
659b66cba7f2d24917e1788e
Yonglong Tian
admin_assigned
2024-01-08T11:41:55.267Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
659b66cba7f2d24917e1788f
Yue Wang
Denoising Vision Transformers
We delve into a nuanced but significant challenge inherent to Vision
Transformers (ViTs): feature maps of these models exhibit grid-like artifacts,
which detrimentally hurt the performance of ViTs in downstream tasks. Our
investigations trace this fundamental issue down to the positional embeddings
at the input stage. To address this, we propose a novel noise model, which is
universally applicable to all ViTs. Specifically, the noise model dissects ViT
outputs into three components: a semantics term free from noise artifacts and
two artifact-related terms that are conditioned on pixel locations. Such a
decomposition is achieved by enforcing cross-view feature consistency with
neural fields in a per-image basis. This per-image optimization process
extracts artifact-free features from raw ViT outputs, providing clean features
for offline applications. Expanding the scope of our solution to support online
functionality, we introduce a learnable denoiser to predict artifact-free
features directly from unprocessed ViT outputs, which shows remarkable
generalization capabilities to novel data without the need for per-image
optimization. Our two-stage approach, termed Denoising Vision Transformers
(DVT), does not require re-training existing pre-trained ViTs and is
immediately applicable to any Transformer-based architecture. We evaluate our
method on a variety of representative ViTs (DINO, MAE, DeiT-III, EVA02, CLIP,
DINOv2, DINOv2-reg). Extensive evaluations demonstrate that our DVT
consistently and significantly improves existing state-of-the-art
general-purpose models in semantic and geometric tasks across multiple datasets
(e.g., +3.84 mIoU). We hope our study will encourage a re-evaluation of ViT
design, especially regarding the naive use of positional embeddings.
A novel noise model is proposed to eliminate grid-like artifacts in Vision Transformers by decomposing outputs into noise-free and artifact-related components, enhancing performance in downstream tasks.
Vision Transformers ViTs feature maps grid-like artifacts positional embeddings noise model semantics term artifact-related terms cross-view feature consistency neural fields per-image optimization learnable denoiser Denoising Vision Transformers DVT DINO MAE DeiT-III EVA02 CLIP DINOv2 DINOv2-reg semantic tasks geometric tasks mIoU