Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper revisits ridge regression, focusing on optimizing the regularization strength in the fixed-design setting. They derive an iterative algorithm to numerically compute the optimal regularization parameter based on generative parameters, proving convergence under limited noise. Experiments on synthetic data demonstrate that using sample-based parameter estimates with this procedure achieves near-optimal generalization performance across various data regimes with a low computational overhead.
Forget grid search: this iterative method finds near-optimal ridge regression regularization parameters with minimal extra compute.
We consider $L^2$-regularized linear (ridge) regression over a finite data sample $X$ with bounded covariance and linear prediction targets $y$ with additive isotropic noise of finite variance. We present an iterative procedure to compute the optimal regularization strength numerically from the generative parameters in the fixed-$X$ setting and prove its convergence at limited noise levels. Our experimental evaluation over synthetic data shows that the proposed procedure combined with sample-based parameter estimates attains near-optimal random-$X$ generalization across a wide range of sample sizes, aspect ratios, and noise levels, at an added computational cost equivalent to one preliminary ridge regression in the underparameterized regime and two in the overparameterized case.