No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"
New methods designed to preserve data privacy require careful scrutiny.
Failure to preserve privacy is hard to detect, and yet can lead to catastrophic
results when a system implementing a ``privacy-preserving'' method is attacked.
A recent work selected for an Outstanding Paper Award at ICML 2022 (Dong et
al., 2022) claims that dataset condensation (DC) significantly improves data
privacy when training machine learning models. This claim is supported by
theoretical analysis of a specific dataset condensation technique and an
empirical evaluation of resistance to some existing membership inference
attacks.
  In this note we examine the claims in the work of Dong et al. (2022) and
describe major flaws in the empirical evaluation of the method and its
theoretical analysis. These flaws imply that their work does not provide
statistically significant evidence that DC improves the privacy of training ML
models over a naive baseline. Moreover, previously published results show that
DP-SGD, the standard approach to privacy preserving ML, simultaneously gives
better accuracy and achieves a (provably) lower membership attack success rate.