Search papers, labs, and topics across Lattice.
This paper establishes a convergence framework for gradient descent (GD) dynamics applicable to a wide range of neural network architectures beyond the neural tangent kernel (NTK) regime. By proving that GD with regular learning rates converges to a stationary point for almost all initializations, the authors utilize an iterate-dependent PL-type inequality and Lipschitz smoothness along the GD trajectory. The findings reveal that the learning rate scale is influenced more by depth and effective bottleneck dimensions than by width, providing new insights into the training dynamics of modern neural networks.
Gradient descent can reliably converge to stationary points in complex neural networks, challenging previous assumptions about initialization and architecture.
Training dynamics is central to understanding neural networks, yet its theoretical analysis remains difficult even for simple architectures and becomes substantially more challenging for general modern architectures. In this paper, we propose a convergence framework for analyzing gradient descent (GD) dynamics under a broad family of neural network architectures and datasets beyond the neural tangent kernel (NTK) regime. The framework is formulated at the level of network blocks and covers architectures including pre-normalized multi-layer transformers. More precisely, under mild assumptions, we prove that for almost all initializations, GD with regular learning rates converges to the neighbourhood of a stationary point. This is mainly proved by establishing an iterate-dependent PL-type inequality through analyticity and measure-zero arguments, and by proving Lipschitz smoothness along the GD trajectory through polynomial generalized smoothness and a local relaxed dissipative condition. We further interpret the theorem under Xavier initialization and practical architectural scaling, showing that the learning rate scale depends on the depth and effective bottleneck dimensions rather than the largest width. Finally, we derive structural nondegeneracy implications for residual connections and function composition, and provide a generic characterization of global minimizers within our framework.
