Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper introduces randomized momentum transport, a technique that reduces the computational cost of momentum methods by probabilistically evaluating gradient transport corrections. By employing importance weighting and a cheap predictor, the method preserves the conditional mean of the momentum update while introducing a controlled increase in variance. Under various assumptions and for different linear minimization oracle updates, the approach achieves an $O(T^{-1/3})$ stationarity bound with reduced oracle calls, demonstrating significant empirical improvements in training loss and test accuracy on benchmark datasets.
Halving oracle calls in momentum methods can slash training loss by up to 43% without sacrificing convergence.
Momentum methods such as STORM and second-order momentum track the gradient by adding a transport correction to momentum, at the cost of an extra oracle call per iteration. We introduce randomized momentum transport, which evaluates the correction with probability $q$ and otherwise uses a cheap prediction, with importance weighting that preserves the conditional mean and adds $(q^{-1}-1)$ times the prediction loss to the variance. Under transport-bias, second-moment, and residual-control assumptions, linear minimization oracle (LMO) updates, including normalized, sign, and exact spectral (Muon) directions, retain an $O(T^{-1/3})$ stationarity bound for each fixed $q>0$ with a single-gradient initialization. For unnormalized STORM, some $q<1$ lowers the leading coefficient of our sufficient gradient budget if and only if the gradient-difference noise modulus exceeds the smoothness constant, by at most about $10\%$. An online diagonal predictor admits a regret-to-convergence guarantee. At approximately equal oracle cost, querying a quarter of the corrections lowers the final training loss by $18$ to $43\%$ on MNIST1D and raises the test accuracy of Hessian-vector transport on CIFAR-10 by $1.5$ points.