MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised
  Learning of Motion and Content Features
Self-supervised learning of visual representations has been focusing on
learning content features, which do not capture object motion or location, and
focus on identifying and differentiating objects in images and videos. On the
other hand, optical flow estimation is a task that does not involve
understanding the content of the images on which it is estimated. We unify the
two approaches and introduce MC-JEPA, a joint-embedding predictive architecture
and self-supervised learning approach to jointly learn optical flow and content
features within a shared encoder, demonstrating that the two associated
objectives; the optical flow estimation objective and the self-supervised
learning objective; benefit from each other and thus learn content features
that incorporate motion information. The proposed approach achieves performance
on-par with existing unsupervised optical flow benchmarks, as well as with
common self-supervised learning approaches on downstream tasks such as semantic
segmentation of images and videos.
64bf8f8e4561d0aca2e04d49
Adrien Bardes
64bf8f8e4561d0aca2e04d4a
Jean Ponce
64bf8f8e4561d0aca2e04d4b
Yann LeCun
admin_assigned
2023-11-23T11:41:25.445Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised
  Learning of Motion and Content Features
Self-supervised learning of visual representations has been focusing on
learning content features, which do not capture object motion or location, and
focus on identifying and differentiating objects in images and videos. On the
other hand, optical flow estimation is a task that does not involve
understanding the content of the images on which it is estimated. We unify the
two approaches and introduce MC-JEPA, a joint-embedding predictive architecture
and self-supervised learning approach to jointly learn optical flow and content
features within a shared encoder, demonstrating that the two associated
objectives; the optical flow estimation objective and the self-supervised
learning objective; benefit from each other and thus learn content features
that incorporate motion information. The proposed approach achieves performance
on-par with existing unsupervised optical flow benchmarks, as well as with
common self-supervised learning approaches on downstream tasks such as semantic
segmentation of images and videos.
MC-JEPA, a joint-embedding predictive architecture, simultaneously learns optical flow and content features through self-supervised learning, improving both optical flow estimation and downstream tasks like semantic segmentation.
self-supervised learning visual representations content features object motion optical flow estimation joint-embedding predictive architecture shared encoder semantic segmentation