Search papers, labs, and topics across Lattice.
This paper introduces a Gaussian approximation to the finite-sample distribution of the ridge regression estimator, addressing the trade-off between bias and variance in estimation and prediction error. By employing nonstandard asymptotics, the authors allow the regularization parameter to grow with the sample size while accommodating general heteroskedasticity and autocorrelation. The proposed approximation leads to two novel strategies for selecting the regularization parameter, aimed at minimizing average or worst-case excess prediction risk.
A new Gaussian approximation reveals that ridge regression can be optimized for finite samples, leading to improved parameter selection strategies that significantly enhance prediction accuracy.
We present a simple Gaussian approximation to the finite-sample distribution of the classical ridge regression estimator. Our approximation captures the fact that, in finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error. Our approximation is based on nonstandard asymptotics where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size; and $ii)$ we treat the population regression coefficients as \emph{local} to the reference vector that defines the estimator's direction of shrinkage. In contrast to other asymptotic approximations in the literature, we allow for general forms of heteroskedasticity and autocorrelation in the data generating process (at the cost of considering a low-dimensional model where the number of covariates is not allowed to grow with the sample size). We use our simple Gaussian approximation to propose two new strategies to select the regularization parameter for the ridge regression estimator. The suggested strategies select the regularization parameter to minimize either average or worst-case excess prediction risk, where risk is computed using our suggested Gaussian approximation.
