NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
  Patch Neighbor Consistency
We propose sorting patch representations across views as a novel
self-supervised learning signal to improve pretrained representations. To this
end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that
enforces patch-level nearest neighbor consistency across a student and teacher
model, relative to reference batches. Our method leverages a differentiable
sorting method applied on top of pretrained representations, such as
DINOv2-registers to bootstrap the learning signal and further improve upon
them. This dense post-pretraining leads to superior performance across various
models and datasets, despite requiring only 19 hours on a single GPU. We
demonstrate that this method generates high-quality dense feature encoders and
establish several new state-of-the-art results: +5.5% and + 6% for
non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and
+7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.
66c5b3eb2d2fcb957116b49e
Valentinos Pariza
claimed_verified
2024-08-27T07:44:06.788Z
63f79b1bfd6949669625f54e
/avatars/04f7a3a9e71a656ea64aed1bb01db5b1.svg
Valentinos Pariza
user
valentinospariza
66c5b3eb2d2fcb957116b49f
Mohammadreza Salehi
claimed_verified
2024-08-29T08:57:40.286Z
66990ec3aa500cd99d640a19
/avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
Mohammadreza Salehi
user
SMSD75
66c5b3eb2d2fcb957116b4a0
Gertjan Burghouts
66c5b3eb2d2fcb957116b4a1
Francesco Locatello
66c5b3eb2d2fcb957116b4a2
Yuki M. Asano
claimed_verified
2024-10-15T09:11:34.944Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
  Patch Neighbor Consistency
We propose sorting patch representations across views as a novel
self-supervised learning signal to improve pretrained representations. To this
end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that
enforces patch-level nearest neighbor consistency across a student and teacher
model, relative to reference batches. Our method leverages a differentiable
sorting method applied on top of pretrained representations, such as
DINOv2-registers to bootstrap the learning signal and further improve upon
them. This dense post-pretraining leads to superior performance across various
models and datasets, despite requiring only 19 hours on a single GPU. We
demonstrate that this method generates high-quality dense feature encoders and
establish several new state-of-the-art results: +5.5% and + 6% for
non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and
+7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.
NeCo, a novel self-supervised learning method using patch neighbor consistency, improves pretrained representations with minimal computational cost and achieves state-of-the-art results in semantic segmentation.
self-supervised learning patch neighbor consistency NeCo training loss student model teacher model reference batches differentiable sorting DINOv2 dense feature encoders semantic segmentation ADE20k Pascal VOC COCO-Things COCO-Stuff