Search papers, labs, and topics across Lattice.
This paper investigates Adam's convergence properties on highly degenerate polynomials, identifying conditions for local asymptotic stability without external schedulers or specific hyperparameter tuning. The authors theoretically prove and empirically validate that Adam achieves local linear convergence on these functions, outperforming Gradient Descent and Momentum due to a decoupling mechanism between the second moment estimate and the squared gradient. Furthermore, they characterize Adam's hyperparameter phase diagram, revealing stable convergence, spikes, and SignGD-like oscillation regimes.
Adam can achieve linear convergence on highly degenerate polynomials without careful tuning, thanks to a built-in mechanism that exponentially amplifies the effective learning rate.
Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the "natural" auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.
