Search papers, labs, and topics across Lattice.
This paper provides the first unconditional error analysis for the Adam optimizer when applied to strongly convex stochastic optimization problems (SOPs). The key contribution is establishing uniform a priori bounds for Adam, removing the previously required assumption that Adam remains uniformly bounded and does not diverge. This allows for a complete error analysis of Adam without conditional convergence assumptions.
The mystery of whether Adam can provably converge on strongly convex problems is finally solved with the first unconditional error analysis.
The adaptive moment estimation (Adam) optimizer proposed by Kingma&Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.
