Rethinking PCA Through Duality
Jan Quan, Johan Suykens, Panagiotis Patrinos
Motivated by the recently shown close connection between self-attention and (kernel) principal component analysis (PCA), we revisit PCA through the framework of difference-of-convex (DC) duality, presenting several novel formulations and providing new theoretical insights. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we develop new algorithms for PCA and empirically compare them with state-of-the-art approaches. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the $l_1$ deviation of the reconstruction errors.