Predicting masked tokens in stochastic locations improves masked image
  modeling
Self-supervised learning is a promising paradigm in deep learning that
enables learning from unlabeled data by constructing pretext tasks that require
learning useful representations. In natural language processing, the dominant
pretext task has been masked language modeling (MLM), while in computer vision
there exists an equivalent called Masked Image Modeling (MIM). However, MIM is
challenging because it requires predicting semantic content in accurate
locations. E.g, given an incomplete picture of a dog, we can guess that there
is a tail, but we cannot determine its exact location. In this work, we propose
FlexPredict, a stochastic model that addresses this challenge by incorporating
location uncertainty into the model. Specifically, we condition the model on
stochastic masked token positions to guide the model toward learning features
that are more robust to location uncertainties. Our approach improves
downstream performance on a range of tasks, e.g, compared to MIM baselines,
FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for
semi-supervised video segmentation using ViT-L.
64c9c380bd5f839f2fe4b198
Amir Bar
64c9c380bd5f839f2fe4b199
Florian Bordes
admin_assigned
2023-08-02T08:09:12.241Z
63166336894404e2506b8811
/avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
Florian Bordes
user
Fbordes
64c9c380bd5f839f2fe4b19a
Assaf Shocher
64c9c380bd5f839f2fe4b19b
Mahmoud Assran
64c9c380bd5f839f2fe4b19c
Pascal Vincent
64c9c380bd5f839f2fe4b19d
Nicolas Ballas
64c9c380bd5f839f2fe4b19e
Trevor Darrell
64c9c380bd5f839f2fe4b19f
Amir Globerson
admin_assigned
2023-08-02T08:09:45.848Z
62cd956bc589e4a9e23ea347
/avatars/dbdccd9258c9d18173d312a8bf14dd6e.svg
Amir Globerson
user
amirgloberson
64c9c380bd5f839f2fe4b1a0
Yann LeCun
admin_assigned
2023-11-23T11:40:45.141Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
Predicting masked tokens in stochastic locations improves masked image
  modeling
Self-supervised learning is a promising paradigm in deep learning that
enables learning from unlabeled data by constructing pretext tasks that require
learning useful representations. In natural language processing, the dominant
pretext task has been masked language modeling (MLM), while in computer vision
there exists an equivalent called Masked Image Modeling (MIM). However, MIM is
challenging because it requires predicting semantic content in accurate
locations. E.g, given an incomplete picture of a dog, we can guess that there
is a tail, but we cannot determine its exact location. In this work, we propose
FlexPredict, a stochastic model that addresses this challenge by incorporating
location uncertainty into the model. Specifically, we condition the model on
stochastic masked token positions to guide the model toward learning features
that are more robust to location uncertainties. Our approach improves
downstream performance on a range of tasks, e.g, compared to MIM baselines,
FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for
semi-supervised video segmentation using ViT-L.
FlexPredict enhances self-supervised learning in image processing by incorporating stochastic location uncertainty into masked image modeling, leading to improved performance on tasks like linear probing and video segmentation.
self-supervised learning pretext tasks masked language modeling masked image modeling FlexPredict stochastic model location uncertainty stochastic masked token positions downstream performance ImageNet linear probing semi-supervised video segmentation ViT-B ViT-L