Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FOAM, an adaptive damping method designed to address the staleness-oriented errors in the Shampoo optimization algorithm, which is hindered by the computational overhead of matrix inversion. By theoretically analyzing the trade-offs between convergence, stability, and computational efficiency, the authors establish that dynamic damping can mitigate the adverse effects of stale updates. Experimental results reveal that FOAM not only accelerates training time compared to traditional Shampoo but also preserves convergence quality, highlighting its practical utility in large-scale optimization tasks.
Dynamic damping in FOAM cuts training time while maintaining robust convergence, challenging the traditional trade-offs in optimization algorithms.
Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion. To mitigate this, practitioners typically rely on stale preconditioner updates, creating a fundamental trade-off between computational efficiency and optimization fidelity. In this work, we provide a theoretical study of staleness through the complementary lenses of convergence and stability. While staleness improves computational efficiency, it inherently degrades performance and introduces numerical instability. Crucially, we identify that damping, acting as a numerical stabilizer, can effectively suppress these negative effects. Guided by this analysis, we propose FOAM, an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency based on an approximation of the staleness-oriented error. Experimental results demonstrate that FOAM reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.