Search papers, labs, and topics across Lattice.
This paper introduces a family of generalized Hessian estimators using random direction stochastic approximation (RDSA) that rely solely on noisy function measurements. The key innovation lies in demonstrating that Hessian estimators utilizing more function measurements exhibit lower-order estimation bias, leading to improved accuracy. The authors provide both asymptotic and non-asymptotic convergence analyses for stochastic Newton methods employing these generalized Hessian estimators, and validate their theoretical results through numerical experiments.
More function measurements in random direction stochastic approximation can significantly reduce estimation bias in Hessian estimation, improving the performance of stochastic Newton methods.
We present a family of generalized Hessian estimators of the objective using random direction stochastic approximation (RDSA) by utilizing only noisy function measurements. The form of each estimator and the order of the bias depend on the number of function measurements. In particular, we demonstrate that estimators with more function measurements exhibit lower-order estimation bias. We show the asymptotic unbiasedness of the estimators. We also perform asymptotic and non-asymptotic convergence analyses for stochastic Newton methods that incorporate our generalized Hessian estimators. Finally, we perform numerical experiments to validate our theoretical findings.
