Cluster and Predict Latents Patches for Improved Masked Image Modeling
Masked Image Modeling (MIM) offers a promising approach to self-supervised
representation learning, however existing MIM models still lag behind the
state-of-the-art. In this paper, we systematically analyze target
representations, loss functions, and architectures, to introduce CAPI - a novel
pure-MIM framework that relies on the prediction of latent clusterings. Our
approach leverages a clustering-based loss, which is stable to train, and
exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
substantially outperforming previous MIM methods and approaching the
performance of the current state-of-the-art, DINOv2. We release all our code
and models.
67b32a554d60b7d162dffd89
Timothée Darcet
claimed_verified
2025-03-03T11:15:21.470Z
63047da67424d937fa31b2b0
/avatars/cfdbb2054da0af37834a619f6c03db52.svg
Timothée Darcet
user
TimDarcet
67b32a554d60b7d162dffd8a
Federico Baldassarre
67b32a554d60b7d162dffd8b
Maxime Oquab
67b32a554d60b7d162dffd8c
Julien Mairal
67b32a554d60b7d162dffd8d
Piotr Bojanowski
Cluster and Predict Latents Patches for Improved Masked Image Modeling
Masked Image Modeling (MIM) offers a promising approach to self-supervised
representation learning, however existing MIM models still lag behind the
state-of-the-art. In this paper, we systematically analyze target
representations, loss functions, and architectures, to introduce CAPI - a novel
pure-MIM framework that relies on the prediction of latent clusterings. Our
approach leverages a clustering-based loss, which is stable to train, and
exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
substantially outperforming previous MIM methods and approaching the
performance of the current state-of-the-art, DINOv2. We release all our code
and models.
Masked Image Modeling (MIM) self-supervised representation learning target representations loss functions architectures pure-MIM framework latent clusterings clustering-based loss scaling properties ViT-L backbone ImageNet mIoU ADE20K linear probes DINOv2