The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions
We introduce dataset multiplicity, a way to study how inaccuracies,
uncertainty, and social bias in training datasets impact test-time predictions.
The dataset multiplicity framework asks a counterfactual question of what the
set of resultant models (and associated test-time predictions) would be if we
could somehow access all hypothetical, unbiased versions of the dataset. We
discuss how to use this framework to encapsulate various sources of uncertainty
in datasets' factualness, including systemic social bias, data collection
practices, and noisy labels or features. We show how to exactly analyze the
impacts of dataset multiplicity for a specific model architecture and type of
uncertainty: linear models with label errors. Our empirical analysis shows that
real-world datasets, under reasonable assumptions, contain many test samples
whose predictions are affected by dataset multiplicity. Furthermore, the choice
of domain-specific dataset multiplicity definition determines what samples are
affected, and whether different demographic groups are disparately impacted.
Finally, we discuss implications of dataset multiplicity for machine learning
practice and research, including considerations for when model outcomes should
not be trusted.