Search papers, labs, and topics across Lattice.
This paper addresses the limitations of traditional stopping criteria in stochastic gradient descent (SGD) by developing trajectory-adaptive stopping rules that leverage the evolving optimization trajectory. By constructing upper confidence sequences that adapt to realized stochastic gradients, the authors achieve optimal decay rates for suboptimality while allowing for early stopping based on certified accuracy. Experimental results demonstrate that these adaptive rules can significantly reduce the number of iterations required compared to conventional deterministic horizons.
Trajectory-adaptive stopping rules can cut SGD iterations by several orders of magnitude while maintaining statistical validity and optimal decay rates.
Stochastic gradient descent (SGD) is typically analyzed at a deterministic horizon chosen before the algorithm is run, even though practical stopping decisions are made adaptively by inspecting the evolving trajectory. This mismatch creates a fundamental certification problem: fixed-time guarantees do not generally remain valid at data-dependent stopping times, while deterministic horizons derived from worst-case bounds can be highly conservative. We address this problem for strongly convex stochastic optimization by constructing fully observable, trajectory-adaptive upper confidence sequences for the squared distance of the last iterate to the optimizer and the suboptimality of a weighted average. These bounds hold simultaneously over time, attain the optimal $1/t$ decay rate up to iterated-logarithmic factors in the worst case, and adapt to the realized stochastic gradients, allowing SGD to stop as soon as a prescribed accuracy is certified without sacrificing statistical validity. Our approach treats the evolving SGD trajectory as a sequential experiment whose observations provide evidence about the unknown optimization error. To formalize this perspective, we develop new recursive confidence-sequence techniques and a general time-uniform empirical Bernstein inequality for adapted processes with time-varying conditional means and predictable ranges that may grow without bound. We further extend these confidence-sequence constructions to minibatch SGD, with the empirical Bernstein bounds exploiting the realized second-moment structure within each minibatch. Numerical experiments show that the resulting stopping rules can require several orders of magnitude fewer iterations than natural deterministic horizons.
