Make Every Example Count: On Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets
Increasingly larger datasets have become a standard ingredient to advancing
the state of the art in NLP. However, data quality might have already become
the bottleneck to unlock further gains. Given the diversity and the sizes of
modern datasets, standard data filtering is not straight-forward to apply,
because of the multifacetedness of the harmful data and elusiveness of
filtering rules that would generalize across multiple tasks. We study the
fitness of task-agnostic self-influence scores of training examples for data
cleaning, analyze their efficacy in capturing naturally occurring outliers, and
investigate to what extent self-influence based data cleaning can improve
downstream performance in machine translation, question answering and text
classification, building up on recent approaches to self-influence calculation
and automated curriculum learning.