Search papers, labs, and topics across Lattice.
This paper establishes a unified framework to explore conservation laws in modern neural architectures, extending the understanding of gradient descent dynamics beyond traditional linear and ReLU networks. By analyzing various activation functions and architectures, including GELU, SiLU, SwiGLU, and multihead attention with different positional encodings, the authors uncover implicit biases that govern model behavior during training. Experimental validation of these theoretical predictions highlights the significance of these conservation laws in explaining the success of over-parameterized models.
Conservation laws reveal how modern neural architectures maintain implicit biases during training, shedding light on their remarkable performance.
Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.
