Search papers, labs, and topics across Lattice.
This paper investigates transfer learning for linear regression using multiple potentially overparameterized least-squares pretrained models, formulating the target learning task as a penalized optimization problem minimizing squared errors on the target dataset. The authors derive an analytical expression for the test error of the learned target model, demonstrating that using more pretrained models improves transfer learning, especially when debiasing overparameterized models. They propose a multiplicative correction factor to mitigate overparameterization bias, enabling the effective use of more pretrained models.
Overparameterized pretrained models can hurt transfer learning in linear regression, but a simple debiasing technique unlocks the benefits of using many such models.
We study transfer learning for a linear regression task using several least-squares pretrained models that can be overparameterized. We formulate the target learning task as optimization that minimizes squared errors on the target dataset with penalty on the distance of the learned model from the pretrained models. We analytically formulate the test error of the learned target model and provide the corresponding empirical evaluations. Our results elucidate when using more pretrained models can improve transfer learning. Specifically, if the pretrained models are overparameterized, using sufficiently many of them is important for beneficial transfer learning. However, the learning may be compromised by overparameterization bias of pretrained models, i.e., the minimum $\ell_2$-norm solution's restriction to a small subspace spanned by the training examples in the high-dimensional parameter space. We propose a simple debiasing via multiplicative correction factor that can reduce the overparameterization bias and leverage more pretrained models to learn a target predictor.
