Dataset Management Platform for Machine Learning
The quality of the data in a dataset can have a substantial impact on the
performance of a machine learning model that is trained and/or evaluated using
the dataset. Effective dataset management, including tasks such as data
cleanup, versioning, access control, dataset transformation, automation,
integrity and security, etc., can help improve the efficiency and speed of the
machine learning process. Currently, engineers spend a substantial amount of
manual effort and time to manage dataset versions or to prepare datasets for
machine learning tasks. This disclosure describes a platform to manage and use
datasets effectively. The techniques integrate dataset management and dataset
transformation mechanisms. A storage engine is described that acts as a source
of truth for all data and handles versioning, access control etc. The dataset
transformation mechanism is a key part to generate a dataset (snapshot) to
serve different purposes. The described techniques can support different
workflows, pipelines, or data orchestration needs, e.g., for training and/or
evaluation of machine learning models.