Search papers, labs, and topics across Lattice.
This paper introduces a unified model for understanding the abrupt learning dynamics of neural networks, revealing that despite diverse architectures, training behaviors can be captured by a common quadratic form. The authors leverage symmetry principles to derive a universal leading expansion that encapsulates the training dynamics of various architectures, including perceptrons and attention layers, through a single structure matrix. Key findings indicate that the training dynamics can be described by a Lotka鈥揤olterra equation, with the model predicting power law behaviors in training time as a function of initial weight sizes.
A universal quadratic model reveals that diverse neural architectures share a common training dynamic, leading to predictable power law behaviors in learning.
Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training losses instead follow smooth power laws. Variants of both behaviors occur in architectures with very different microscopic structures, which is the signature of a few relevant collective variables. We show that a symmetry fixes what those variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training, the quadratic $\Tr[WW^{\top}A(x)]$, in which every architectural detail is confined to a single ``structure matrix"$A(x)$ that we compute for each architecture. Perceptrons, attention layers, mixtures of experts, and convolutions become one model at different $A$. Its training dynamics then close on the ``order parameter"$M=WW^{\top}$ and, whenever the data matrices share an eigenbasis, reduce to a Lotka--Volterra equation whose modes switch on one after another. The smaller the initial weights, the further apart the switch-on times, and the plateaus appear as a singular limit of a smooth flow; when many modes are unresolved the same events merge into a power law in training time whose exponent the theory predicts. We confirm both numerically across training methods and architectures.
MIT CSAIL