Probing the Transition to Dataset-Level Privacy in ML Models Using an Output-Specific and Data-Resolved Privacy Profile
Differential privacy (DP) is the prevailing technique for protecting user
data in machine learning models. However, deficits to this framework include a
lack of clarity for selecting the privacy budget $\epsilon$ and a lack of
quantification for the privacy leakage for a particular data row by a
particular trained model. We make progress toward these limitations and a new
perspective by which to visualize DP results by studying a privacy metric that
quantifies the extent to which a model trained on a dataset using a DP
mechanism is ``covered" by each of the distributions resulting from training on
neighboring datasets. We connect this coverage metric to what has been
established in the literature and use it to rank the privacy of individual
samples from the training set in what we call a privacy profile. We
additionally show that the privacy profile can be used to probe an observed
transition to indistinguishability that takes place in the neighboring
distributions as $\epsilon$ decreases, which we suggest is a tool that can
enable the selection of $\epsilon$ by the ML practitioner wishing to make use
of DP.