Adversarial Self-Supervised Representation Learning with Theoretical Guarantees
Chenguang Duan, Yuling Jiao, Huazhen Lin, Wensen Ma, Jerry Yang
Learning transferable data representations from abundant unlabeled data remains a critical challenge in machine learning. While numerous self-supervised learning methods have emerged to address this challenge, a significant class of these approaches aligns the covariance or correlation matrix with the identity matrix. Despite impressive performance across various downstream tasks, these methods often suffer from biased sample risk, leading to significant optimization shifts in mini-batch settings and hindering theoretical analysis. In this paper, we introduce a novel Adversarial Self-Supervised Representation Learning (ASSRL) for unbiased transfer learning. Our method not only outperforms existing methods across multiple benchmark datasets but also offers comprehensive end-to-end theoretical guarantees. Our analysis reveals that minimax optimization drives representations to form well-separated clusters in the embedding space, provided sufficient upstream unlabeled data. Consequently, our method delivers strong classification performance even with limited downstream labels, shedding new light on few-shot learning.