Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Yeichan Kim, Ilmun Kim, Seyoung Park
Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-$\ell_2$-norm interpolator (MNI) in high-dimensional linear regression have garnered much attention for their remarkable generalization capabilities, a property termed *benign overfitting*. Despite their individual importance, the intersection of transfer learning and the MNI is largely unexplored. Our research bridges this gap by proposing a novel two-step transfer MNI approach and analyzing its trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals a *free-lunch* covariate shift, where leveraging heterogeneous data offers the benefit of knowledge transfer with limited cost. To operationalize our findings, we develop a data-driven procedure to detect *informative* sources and propose an ensemble method incorporating multiple transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.