Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
We introduce a second-order recurrent model that replaces standard neuron-to-neuron communication with a spatially evolving field governed by discretized partial differential equations, inspired by cortical waves. This mechanism enables structured spatiotemporal patterns to function as an implicit, high-capacity memory, rendering the model equivalent to a structured infinite-order RNN with an effectively unbounded receptive field and fixed parameters. Empirically, this architecture achieves superior performance on long-horizon tasks compared to other recurrent models with significantly fewer parameters, demonstrating the efficacy of spatial dynamics for bridging efficient inference and long-term memory.
Recurrent models can achieve infinite context windows by replacing scalar neuron states with evolving spatial fields governed by PDEs, effectively creating an implicit infinite-order RNN.
Recurrent neural networks (RNNs) offer linear-time scaling with sequence length while requiring only constant memory, yet they struggle to capture long-range dependencies due to vanishing gradients and limited receptive fields. To address these limitations, we introduce a second-order recurrent model in which the standard neuron-to-neuron communication is replaced by a spatially evolving field governed by (discretized) partial differential equations. Drawing inspiration from the role of cortical waves in brain computation, this mechanism allows structured spatiotemporal patterns to serve as an implicit, high-capacity memory. We show that the resulting model is equivalent to a structured infinite-order RNN in which the current state depends explicitly on its entire history of past states, yielding an effectively unbounded receptive field with a fixed number of parameters. We further derive constructive conditions to ensure marginal stability, constraining the gradient spectrum on the unit circle and thereby eliminating vanishing and exploding gradients. Empirically, the proposed architecture outperforms other recurrent models on long-horizon benchmarks while using substantially fewer parameters, demonstrating that spatial dynamics can effectively bridge the gap between efficient inference and long-term memory.