Search papers, labs, and topics across Lattice.
This study treats the training trajectories of neural networks as temporal networks and applies scalar embeddings to analyze their dynamics. By focusing on a multilayer perceptron trained on the MNIST dataset, the authors demonstrate that these embeddings effectively capture key dynamical features, such as sensitivity to initial conditions and the maximum Lyapunov exponent. The findings reveal a characteristic time that reflects the decorrelation of trajectories, suggesting that scalar representations can enhance our understanding of neural network optimization processes.
Scalar embeddings reveal that neural network training dynamics can be effectively summarized, preserving critical features like sensitivity to initial conditions and decorrelation times.
Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging. In this work, we treat such training trajectories as temporal networks and apply recently proposed strategies for the scalar embedding of temporal networks. We investigate whether such a scalar embedding provides a meaningful low-dimensional representation of neural network training dynamics. Using a multilayer perceptron trained on the MNIST classification task, we show that the embedding preserves the main dynamical features observed in the original parameter space, including the emergence of sensitivity to initial conditions for specific learning rate regimes and an accurate reconstruction of the network's maximum Lyapunov exponent. We then use the embedded scalar trajectory to define a characteristic time, analogous to a Lyapunov time, after which the exponential separation between initially close embedded trajectories saturates. This characteristic time captures the typical decorrelation time between initially close network trajectories in the original high-dimensional system. Finally, we investigate the statistical organization of asymptotic training states through a spacing observable defined in the embedded space. We find that the distributions of rescaled asymptotic spacings collapse onto a common form across initial conditions and are compatible with a skew lognormal distribution. Altogether, our results suggest that scalar low-dimensional embeddings provide a useful framework for studying and visualizing the dynamical properties of neural network optimization trajectories.
