Search papers, labs, and topics across Lattice.
This paper introduces a novel, fully online de-biased covariance estimator for stochastic gradient descent (SGD) that avoids the need for Hessian information. By employing a bias-reduction technique, the proposed estimator achieves a convergence rate of $n^{(\alpha-1)/2} \sqrt{\log n}$, surpassing existing Hessian-free methods in accuracy. The improved covariance estimation is crucial for reliable online inference and uncertainty quantification in SGD-based learning.
Forget slow convergence and inaccessible Hessians: this new de-biased covariance estimator turbocharges SGD with faster, more accurate uncertainty estimates.
We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(\alpha-1)/2} \sqrt{\log n}$, outperforming existing Hessian-free alternatives.
