Search papers, labs, and topics across Lattice.
The paper introduces the Neuronal Stochastic Attention Circuit (NSAC), a novel continuous-time attention mechanism that models attention logits as the solution to an Ornstein-Uhlenbeck stochastic differential equation modulated by C. elegans-inspired neuronal circuits. This induces a Gaussian distribution over logits, enabling probabilistic attention weights and output, and allowing for the quantification of both aleatoric and epistemic uncertainty. Experiments across various tasks, including time-series forecasting and autonomous driving, demonstrate that NSAC achieves competitive accuracy while providing well-calibrated uncertainty estimates.
By reformulating attention as a stochastic differential equation modulated by biologically-inspired neuronal circuits, this work offers a new path to interpretable and well-calibrated uncertainty estimates in continuous-time representation learning.
Reliable quantification of uncertainty estimates in continuous-time (CT) representation learning remains nascent, particularly within CT attention architectures. We introduce the Neuronal Stochastic Attention Circuit (NSAC), a novel biologically-inspired CT attention architecture that reformulates attention logit computation as the solution of an Ornstein-Uhlenbeck stochastic differential equation modulated by input-dependent, nonlinear interlinked gates derived from repurposed C.elegans Neuronal Circuit Policies (NCPs) wiring mechanism. It induces Gaussian distribution over logits that propagates principled stochasticity through logistic-normal distribution over attention weights to yield probabilistic output. A two-term objective function combining Gaussian negative log-likelihood with an epistemic-separation regularizer enforces higher predictive variance and enables joint quantification of aleatoric and epistemic uncertainty. Empirically, we implement NSAC in a diverse set of learning tasks including: (i) irregular CT function approximation; (ii) multivariate regression; (iii) long-range forecasting; (iv) Industry 4.0; and (v) the lane-keeping of autonomous vehicles. We observe that the NSAC remains competitive against several baselines in terms of accuracy and produces reasonably well-calibrated uncertainty estimates while being interpretable at the neuronal cell level.
