Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance
Differentially private stochastic gradient descent privatizes model training
by injecting noise into each iteration, where the noise magnitude increases
with the number of model parameters. Recent works suggest that we can reduce
the noise by leveraging public data for private machine learning, by projecting
gradients onto a subspace prescribed by the public data. However, given a
choice of public datasets, it is not a priori clear which one may be most
appropriate for the private task. We give an algorithm for selecting a public
dataset by measuring a low-dimensional subspace distance between gradients of
the public and private examples. We provide theoretical analysis demonstrating
that the excess risk scales with this subspace distance. This distance is easy
to compute and robust to modifications in the setting. Empirical evaluation
shows that trained model accuracy is monotone in this distance.