Mitigating Spurious Features in Contrastive Learning with Spectral Regularization
Naghmeh Ghanooni, Dennis Wagner, Waleed Mustafa, Sophie Burkhardt, Anthony Lin, Marius Kloft
Neural networks generally prefer simple and easy to learn features. When these features are spuriously correlated with the labels, the network's performance can suffer, particularly for underrepresented classes or concepts. Self-supervised representation learning methods, such as contrastive learning, are especially prone to this issue, often resulting in worse performance of downstream tasks. We empirically link this limitation to the spectrum of the feature matrix and provide theoretical support of this phenomenon.We propose a novel framework for learning high-rank feature representations, enabling the extraction of more meaningful and diverse features. Our approach achieves this without relying on ground-truth labels or any additional information. We demonstrate the applicability of the proposed framework using SimCLR, a prominent contrastive learning paradigm, and highlight its potential for broader use across representation learning methodologies.