Search papers, labs, and topics across Lattice.
This paper presents a counterexample to the Fourier alignment hypothesis in single-neuron modular addition by demonstrating that an active ReLU neuron can become inactive in finite time, leading to a distribution of Fourier energy across all nonzero real frequency classes. The findings indicate that this phenomenon can occur with positive probability under Gaussian initialization and across a range of initial conditions. Additionally, the results are reinforced by an appendix that extends the counterexample to various conditions, challenging the assumption that single-frequency alignment is a guaranteed outcome of training in this context.
An active ReLU neuron can become completely inactive during training, defying the expectation of Fourier alignment in modular addition tasks.
We give a negative solution to MAIS-O60. We first construct an example in which an initially active ReLU neuron becomes completely inactive in finite time and thereafter remains frozen at a limit whose Fourier energy is equally distributed among all nonzero real frequency classes. The counterexample holds on an open set of initial conditions and therefore occurs with positive probability under Gaussian initialization. An appendix prepared by GPT-5.6 Sol strengthens the counterexample by showing that the same failure can occur for every Clarke trajectory from an open set of initial conditions, under the convention $\mathrm{ReLU}'(0)=0$, for smooth dead-zone approximations of ReLU, and for fixed-step full-batch gradient descent. Thus, single-frequency alignment is not a general consequence of training a single neuron on modular addition.
