Cluster and Predict Latents Patches for Improved Masked Image Modeling
Masked Image Modeling (MIM) offers a promising approach to self-supervised
representation learning, however existing MIM models still lag behind the
state-of-the-art. In this paper, we systematically analyze target
representations, loss functions, and architectures, to introduce CAPI - a novel
pure-MIM framework that relies on the prediction of latent clusterings. Our
approach leverages a clustering-based loss, which is stable to train, and
exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
substantially outperforming previous MIM methods and approaching the
performance of the current state-of-the-art, DINOv2. We release all our code
and models.
67b32a554d60b7d162dffd89
Timothée Darcet
claimed_verified
2025-03-03T11:15:21.470Z
63047da67424d937fa31b2b0
/avatars/cfdbb2054da0af37834a619f6c03db52.svg
Timothée Darcet
user
TimDarcet
67b32a554d60b7d162dffd8a
Federico Baldassarre
extracted_confirmed
2025-05-31T13:57:01.887Z
631e0c1e0db849840ab6bf3e
/avatars/e6d469e804201a6078ccb8369ed97384.svg
Federico
user
baldassarre
67b32a554d60b7d162dffd8b
Maxime Oquab
67b32a554d60b7d162dffd8c
Julien Mairal
67b32a554d60b7d162dffd8d
Piotr Bojanowski
Cluster and Predict Latents Patches for Improved Masked Image Modeling
Masked Image Modeling (MIM) offers a promising approach to self-supervised
representation learning, however existing MIM models still lag behind the
state-of-the-art. In this paper, we systematically analyze target
representations, loss functions, and architectures, to introduce CAPI - a novel
pure-MIM framework that relies on the prediction of latent clusterings. Our
approach leverages a clustering-based loss, which is stable to train, and
exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
substantially outperforming previous MIM methods and approaching the
performance of the current state-of-the-art, DINOv2. We release all our code
and models.
A novel MIM framework CAPI using clustering-based loss achieves high accuracy on ImageNet and ADE20K, outperforming previous methods.
MIM self-supervised representation learning target representations loss functions architectures clustering-based loss ViT-L backbone ImageNet ADE20K DINOv2