Dataset Factory: A Toolchain For Generative Computer Vision Datasets
Generative AI workflows heavily rely on data-centric tasks - such as
filtering samples by annotation fields, vector distances, or scores produced by
custom classifiers. At the same time, computer vision datasets are quickly
approaching petabyte volumes, rendering data wrangling difficult. In addition,
the iterative nature of data preparation necessitates robust dataset sharing
and versioning mechanisms, both of which are hard to implement ad-hoc. To solve
these challenges, we propose a "dataset factory" approach that separates the
storage and processing of samples from metadata and enables data-centric
operations at scale for machine learning teams and individual researchers.