Search papers, labs, and topics across Lattice.
This paper addresses the issue of spectral bias in deep neural networks (DNNs) that hinders their ability to model oscillatory and multiscale behaviors effectively. By analyzing the shortcomings of shallow ReLU networks in fitting high-frequency functions, the authors introduce RepNet, a reparameterized DNN model that optimally adjusts the initial slope scale and partition point distribution. Numerical experiments demonstrate that RepNet significantly enhances the accuracy of DNNs in capturing complex oscillatory features with minimal additional computational cost.
RepNet outperforms traditional DNNs in modeling high-frequency functions while maintaining computational efficiency.
Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNet, a reparameterized DNN model for ReLU and tanh networks designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one- and four-dimensional function approximation, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning, demonstrate that RepNet improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features with slightly additional computational cost. These results indicate that RepNet provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.
