DC-BENCH: Dataset Condensation Benchmark
Dataset Condensation is a newly emerging technique aiming at learning a tiny
dataset that captures the rich information encoded in the original dataset. As
the size of datasets contemporary machine learning models rely on becomes
increasingly large, condensation methods become a prominent direction for
accelerating network training and reducing data storage. Despite numerous
methods have been proposed in this rapidly growing field, evaluating and
comparing different condensation methods is non-trivial and still remains an
open issue. The quality of condensed dataset are often shadowed by many
critical contributing factors to the end performance, such as data augmentation
and model architectures. The lack of a systematic way to evaluate and compare
condensation methods not only hinders our understanding of existing techniques,
but also discourages practical usage of the synthesized datasets. This work
provides the first large-scale standardized benchmark on Dataset Condensation.
It consists of a suite of evaluations to comprehensively reflect the
generability and effectiveness of condensation methods through the lens of
their generated dataset. Leveraging this benchmark, we conduct a large-scale
study of current condensation methods, and report many insightful findings that
open up new possibilities for future development. The benchmark library,
including evaluators, baseline methods, and generated datasets, is open-sourced
to facilitate future research and application.