Progressive Data Dropout: An Embarrassingly Simple Approach to Train Faster
Shriram M S, Xinyue Hao, Shihao Hou, Yang Lu, Laura Sevilla-Lara, Anurag Arnab, Shreyank Gowda
Training deep neural networks is computationally expensive, often requiring significant time and energy to converge on large datasets. In this paper, we propose Progressive Data Dropout, an embarrassingly simple yet effective approach to improve training efficiency by progressively discarding subsets of data across epochs. We explore three variants: (1) a curriculum-inspired method that initially focuses on hard examples—those misclassified or predicted with low confidence; (2) a scalar-based decay method that randomly drops a fixed proportion of data each epoch; and (3) a hybrid approach that mimics the schedule of the first method but drops data at random rather than based on difficulty. All three approaches use the full training dataset only in the last epochs. Despite its simplicity, the third variant achieves the best performance in our experiments. Remarkably, our approach reduces the number of effective epochs to as little as 12.4\% of the baseline measured in 'Effective Epochs', a hardware-independent proxy for backpropagation effort while improving accuracy by up to 4.82\%, depending on the model and dataset. Our approach requires no changes to model architecture or optimizer, and can be applied across standard training pipelines. These results demonstrate that carefully designed data dropout strategies can substantially reduce training costs while enhancing generalization. Code: https://anonymous.4open.science/r/LearningWithRevision-1B1B.