Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper quantifies the shape of learning-rate (LR) transfer curves by deriving two relational equations for the LR gradient: a theoretical formula at the edge of stability and an approximate expression treating the gradient as a hyper-parameter gradient. These equations reveal theoretical lower bounds and conditions for zero gradient in normalization layers, while the approximate expression decomposes the LR gradient layer-wise, identifying key contributors to curve shape. This work provides a foundational understanding of the internal mechanisms governing LR dependence in neural network optimization.
A single LR gradient equation explains the shape of learning-rate transfer curves across diverse settings, revealing layer-specific sensitivities and theoretical bounds.
Understanding the learning-rate (LR) dependence of trained model performance is essential for better optimization of neural networks, but the appropriate size of the LR depends on learning dynamics, architecture, and their hidden biases, and thus our understanding of this complex dependence remains limited. The behavior of the loss curve as a function of the LR has recently been observed in extensive experiments on LR transfer across different settings, such as model sizes, and it is becoming more important to explore methodologies for quantitatively characterizing the shape of this LR curve. In this work, we provide two relational equations that give a good empirical explanation of the gradient of the LR transfer curve, i.e., the LR gradient. One is a theoretical formula that holds at the edge of stability, and the other is an approximate expression obtained by naively treating this gradient as a hyper-parameter gradient. Using these two relational equations in a complementary manner, we show a lower bound on the gradient of the curve and a zero gradient under specific conditions for normalization layers in theory. The approximate expression also applies to a wide range of experimental settings and allows us to decompose the LR gradient into layer-wise components and quantify the strong contribution of specific layers to the curve shape. Thus, this serves as a foundation for better understanding the internal mechanisms of LR dependence.