Search papers, labs, and topics across Lattice.
This paper investigates ridge-regularized log-density-ratio estimation within the Gaussian location model, focusing on two approaches: a variational estimator with an empirical KL log-normalized fit and a spectral estimator based on a continuum of ridge-regularized least-squares problems. By deriving high-dimensional deterministic asymptotic equivalents, the authors reveal that the variational estimator outperforms the spectral estimator in terms of population risk when the number of observations is large, while the spectral estimator is advantageous with fewer observations due to its lower variance. Additionally, the study explores the application of a nuclear penalty for feature learning, providing insights into the trade-offs between the two estimation methods.
In high-dimensional settings, the variational estimator consistently outperforms the spectral estimator when data is abundant, but the latter shines in low-data scenarios due to its reduced variance.
We study ridge-regularized log-density-ratio estimation in the Gaussian location model with a common covariance matrix. By affine invariance, the model is written as q $\sim$ N(0, I), p $\sim$ N($\Delta$, I), with linear features, where $\Delta$ is a mean vector. The variational estimator is the empirical Kullback-Leibler (KL) log-normalized fit with a squared L2-penalty on its nonconstant coefficient, and the spectral estimator recently introduced in [1] replaces a single variational problem by a continuum of ridge-regularized least-squares problems. We derive high-dimensional deterministic asymptotic equivalents when the numbers of observations and dimension tend to infinity with fixed ratios. The regularized variational limit is characterized by a scalar entropy minimization problem derived from the convex-Gaussian-min-max theorem (CGMT), while the regularized spectral limit follows from deterministic equivalents for resolvents of weighted sums of two independent Gaussian sample covariance matrices. We use these formulas to compare population risks, with experiments focused on fixed-signal aspect-ratio sweeps and optimized regularization. Our conclusion is that with many observations, under the criteria and asymptotic regimes analyzed here, the well-specified variational estimator has the smaller risk, while with fewer observations, the spectral estimator is favored because its covariance-based construction has lower variance. We also study how a nuclear penalty can be used and partially analyzed to perform feature learning.
