Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a comprehensive theoretical analysis of how two-layer ReLU neural networks trained via online minibatch SGD prioritize spurious features over true signal features, specifically in the context of the XOR problem. The authors demonstrate that SGD rapidly learns the spurious feature due to optimization dynamics that couple it with the signal feature, leading to a phase transition where the spurious feature inhibits the learning of the signal. Notably, they show that under strong spurious correlations, the spurious feature can dominate learning even at the threshold where the XOR function would typically be learned in isolation.
SGD can learn spurious features exponentially faster than true signals, fundamentally altering our understanding of neural network training dynamics.
Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations. In this work, we provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. We consider data drawn from the high-dimensional Boolean hypercube with a quadratic signal function (namely XOR) and a linear spurious correlation. We show that SGD learns the spurious feature first, and exponentially fast. Moreover, the optimization dynamics couple the spurious and signal features, with a stronger spurious component inhibiting signal feature learning. Our analysis reveals precise phase transitions in the learning dynamics. In the first phase, alignment between the signs of the spurious feature and second-layer weight drives rapid growth of the spurious feature. In the second phase, large majority group margin slows learning and the signal feature remains suppressed. When the spurious correlation is maximally strong, we show theoretically that the spurious feature dominates even at the sample complexity threshold where XOR would be learned in isolation (i.e., if the spurious feature was absent). In contrast, when the correlation strength is constant, we provide preliminary empirical evidence that the model can eventually learn the XOR signal, although the spurious feature is not forgotten.