Search papers, labs, and topics across Lattice.
This paper introduces the AIM framework, which interprets momentum in optimization as a correction driven by residual penalties, clarifying the relationship between update geometry and acceleration in momentum-based optimizers. By separating the mechanisms of residual penalties and objective-related subproblem approximations, the authors propose RADAR, a novel optimizer that enhances update direction and momentum estimation through relativistic adaptive geometry and second-order momentum filtering. Experimental results demonstrate that RADAR consistently outperforms established adaptive optimizers across various tasks, including supervised vision learning and reinforcement learning.
RADAR achieves superior optimization performance by decoupling momentum estimation from update geometry, leading to consistent improvements over traditional adaptive optimizers.
Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood. We develop an $\textbf{A}$DMM-$\textbf{I}$nspired $\textbf{M}$omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates two mechanisms that are usually intertwined in practical optimizers: the residual penalty determines the update geometry, whereas the approximation of the objective-related subproblem determines the acceleration form. Building on AIM, we propose $\textbf{R}$elativistic $\textbf{A}$daptive gradient $\textbf{D}$escent with $\textbf{A}$ccelerated $\textbf{R}$esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.
