MaxSup: Overcoming Representation Collapse in Label Smoothing
Yuxuan Zhou, Heng Li, Zhi-Qi Cheng, Xudong Yan, Yifei Dong, Mario Fritz, Margret Keuper
Label Smoothing (LS) is widely adopted to mitigate over-confidence and improve generalization, yet accumulating evidence reveals two fundamental pathologies. (i) LS contracts feature embeddings into excessively compact class clusters, obscuring fine-grained intra-class structure; (ii) when the prediction is incorrect, LS paradoxically reinforces the wrong class, encouraging unwarranted confidence in errors. We derive an exact decomposition of the LS objective, demonstrating that the additional term decomposes into a regularization component—beneficial exclusively for correct predictions—and an error-amplification component—activated precisely for incorrect predictions. This single mechanism accounts for both representation collapse and mistaken over-confidence. Guided by this analysis, we introduce Max Suppression (MaxSup), a drop-in alternative that penalizes the top-1 logit instead of the ground-truth logit, thereby preserving the beneficial regularization while eliminating error amplification. Experiments on CIFAR-10/100, ImageNet-1k, and several transfer benchmarks show that MaxSup consistently increases intra-class dispersion, sharpens inter-class margins, and improves top-1 accuracy by up to 1.4 percentage points on ImageNet-1k, all with negligible computational overhead. Code and reproducibility scripts are available at https://anonymous.4open.science/r/Maximum-Suppression-Regularization-DB0C.