Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This work establishes a unified geometric framework that formalizes the exact conditions under which parameter-space continuous symmetries induce conservation laws under gradient flow. By introducing the concept of compositional identifiability, the authors formulate an inheritance principle that allows complete characterization of symmetries and conservation laws across successive layers in deep architectures. They validate this framework by deriving the exact invariant quantities governing training dynamics in multi-head attention, grouped-query attention, and deep linear networks.
Symmetries in deep networks don't automatically yield conservation laws under gradient flow; this geometric framework pinpoints exactly when Noether-style invariants hold across layers, completely mapping the conserved quantities of multi-head and grouped-query attention.
Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.