When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations
In machine learning, incorporating more data is often seen as a reliable
strategy for improving model performance; this work challenges that notion by
demonstrating that the addition of external datasets in many cases can hurt the
resulting model's performance. In a large-scale empirical study across
combinations of four different open-source chest x-ray datasets and 9 different
labels, we demonstrate that in 43% of settings, a model trained on data from
two hospitals has poorer worst group accuracy over both hospitals than a model
trained on just a single hospital's data. This surprising result occurs even
though the added hospital makes the training distribution more similar to the
test distribution. We explain that this phenomenon arises from the spurious
correlation that emerges between the disease and hospital, due to
hospital-specific image artifacts. We highlight the trade-off one encounters
when training on multiple datasets, between the obvious benefit of additional
data and insidious cost of the introduced spurious correlation. In some cases,
balancing the dataset can remove the spurious correlation and improve
performance, but it is not always an effective strategy. We contextualize our
results within the literature on spurious correlations to help explain these
outcomes. Our experiments underscore the importance of exercising caution when
selecting training data for machine learning models, especially in settings
where there is a risk of spurious correlations such as with medical imaging.
The risks outlined highlight the need for careful data selection and model
evaluation in future research and practice.