Automatic Data Curation for Self-Supervised Learning: A Clustering-Based
  Approach
Self-supervised features are the cornerstone of modern machine learning
systems. They are typically pre-trained on data collections whose construction
and curation typically require extensive human effort. This manual process has
some limitations similar to those encountered in supervised learning, e.g., the
crowd-sourced selection of data is costly and time-consuming, preventing
scaling the dataset size. In this work, we consider the problem of automatic
curation of high-quality datasets for self-supervised pre-training. We posit
that such datasets should be large, diverse and balanced, and propose a
clustering-based approach for building ones satisfying all these criteria. Our
method involves successive and hierarchical applications of k-means on a
large and diverse data repository to obtain clusters that distribute uniformly
among data concepts, followed by a hierarchical, balanced sampling step from
these clusters. Extensive experiments on three different data domains including
web-based images, satellite images and text show that features trained on our
automatically curated datasets outperform those trained on uncurated data while
being on par or better than ones trained on manually curated data.
6653f376607894ea80490fca
Huy V. Vo
6653f376607894ea80490fcb
Vasil Khalidov
6653f376607894ea80490fcc
Timothée Darcet
admin_assigned
2024-05-27T09:46:20.299Z
63047da67424d937fa31b2b0
/avatars/cfdbb2054da0af37834a619f6c03db52.svg
Timothée Darcet
user
TimDarcet
6653f376607894ea80490fcd
Théo Moutakanni
admin_assigned
2024-05-27T09:46:36.347Z
639111acb73a0f8c02a92844
/avatars/58766701471be66fd784bedd319741a4.svg
Theo Moutakanni
user
TheoM
6653f376607894ea80490fce
Nikita Smetanin
admin_assigned
2024-05-27T09:46:41.954Z
5ecaa091968f6028e0559f80
/avatars/e7e49ebe9d5755d2e976c9553161ce9b.svg
Nikita Smetanin
user
nikitos9000
6653f376607894ea80490fcf
Marc Szafraniec
admin_assigned
2024-05-27T09:46:53.822Z
6479d0aabfaa9e96b8514eef
/avatars/d7aa2d38c81219b9a8af9e331a2916a0.svg
Marc Szafraniec
user
mszafraniec
6653f376607894ea80490fd0
Hugo Touvron
admin_assigned
2024-05-27T09:46:59.632Z
66002a56fc78eace53a1010d
/avatars/330ca2deeca778f1f0e3deb1b80b5049.svg
Hugo Touvron
user
htouvron
6653f376607894ea80490fd1
Camille Couprie
6653f376607894ea80490fd2
Maxime Oquab
6653f376607894ea80490fd3
Armand Joulin
6653f376607894ea80490fd4
Hervé Jégou
6653f376607894ea80490fd5
Patrick Labatut
admin_assigned
2024-05-27T09:47:34.268Z
6446a623cffecebe8b5da0b4
https://cdn-avatars.huggingface.co/v1/production/uploads/6446a623cffecebe8b5da0b4/TTWVF0lQn8dGOFMxZQJH5.jpeg
Patrick Labatut
user
patricklabatut
6653f376607894ea80490fd6
Piotr Bojanowski
Automatic Data Curation for Self-Supervised Learning: A Clustering-Based
  Approach
Self-supervised features are the cornerstone of modern machine learning
systems. They are typically pre-trained on data collections whose construction
and curation typically require extensive human effort. This manual process has
some limitations similar to those encountered in supervised learning, e.g., the
crowd-sourced selection of data is costly and time-consuming, preventing
scaling the dataset size. In this work, we consider the problem of automatic
curation of high-quality datasets for self-supervised pre-training. We posit
that such datasets should be large, diverse and balanced, and propose a
clustering-based approach for building ones satisfying all these criteria. Our
method involves successive and hierarchical applications of k-means on a
large and diverse data repository to obtain clusters that distribute uniformly
among data concepts, followed by a hierarchical, balanced sampling step from
these clusters. Extensive experiments on three different data domains including
web-based images, satellite images and text show that features trained on our
automatically curated datasets outperform those trained on uncurated data while
being on par or better than ones trained on manually curated data.
A clustering-based approach for automatic curation of high-quality datasets for self-supervised pre-training yields features that outperform those from uncurated data and match manually curated data.
self-supervised features k-means clustering-based approach hierarchical applications balanced sampling web-based images satellite images text