Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the relationship between intrinsic symmetries in training data and conserved quantities during the gradient-flow training of neural networks. The authors prove that, generally, these symmetries do not lead to additional integrals of motion when the loss function is analytic and non-polynomial. However, they identify specific scenarios with mean squared error (MSE) loss where data augmentation can generate extra conserved quantities, leveraging a framework based on tensorizable networks to illustrate this phenomenon.
Data symmetries might not enhance neural network training, but MSE loss with data augmentation can unlock unexpected conserved quantities.
We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks. Under the assumption that the loss function is analytic and non-polynomial, we prove that data symmetries generically do not induce any additional integrals of motion. For mean squared error (MSE) loss, on the other hand, there are situations in which data augmentation yields extra conserved quantities. We build a framework, utilizing \emph{tensorizable networks} to describe this phenomenon. Tensorizable networks are a family of architectures whose dependence on parameters and inputs can be separated using an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention.