Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces an orthogonal polynomial dynamics framework to theoretically explain why shallow neural networks exhibit spectral bias, learning low-frequency sinusoidal components first. By analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix derived from a general Fourier series expansion of network outputs, the authors provide an asymptotic explanation for this phenomenon. The framework extends to general activation functions and makes weak assumptions about data distribution, offering a unified theoretical lens for understanding learning bias.
Neural networks preferentially learn low-frequency sinusoidal patterns due to the inherent graded structure of their gradient descent dynamics, explained by an orthogonal polynomial framework.
Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.