Search papers, labs, and topics across Lattice.
This paper investigates how neural networks learn structured internal representations through the group composition task, specifically training a two-layer network to predict the composition of group elements. By analyzing the training dynamics in the Fourier domain, the authors show that the network converges to a single irreducible representation and achieves a rank-one alignment of cross-layer Fourier coefficients. Their findings provide a representation-theoretic perspective on feature learning and reveal a novel low-rank compression phenomenon in matrix-valued group representations, particularly for Abelian groups, where random initialization fosters uniform diversification and rapid convergence rates.
Neural networks can achieve exponential convergence to structured representations, revealing a surprising link between training dynamics and representation theory.
Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.
