PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoders
Ruixin Guo, Ruoming Jin, Xinyu Li, Yang Zhou
Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. However, this success is largely empirical, with limited theoretical justification.  In this paper, we investigate the generalizability -- a theoretical measure of model performance in statistical machine learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, extending the earlier PAC-Bayes bound for multiple linear regression by Shalaeva et al, and provide sufficient conditions for its theoretical convergence. We then adapt this bound to LAEs by showing they can be interpreted as constrained multivariate linear regression on bounded data. To address the computational challenges arising from these constraints, we develop practical methods for optimizing the bound. Experimental results demonstrate that our bound is tight and correlates well with practical evaluation metrics such as Recall@K and NDCG@K.