Search papers, labs, and topics across Lattice.
This paper proves that activation saturation in Neural ODEs (NODEs) with saturating activations structurally limits their dynamics. Specifically, saturation attenuates the input Jacobian, forcing Floquet exponents along periodic orbits into a narrow interval around zero, thereby collapsing the Floquet spectrum. The authors provide a saturation-weighted spectral factorization to refine these bounds and validate their findings numerically on the Stuart-Landau oscillator, explaining the failure of tanh-NODEs on the Morris-Lecar neuron model.
Activation saturation in Neural ODEs acts as a fundamental governor, preventing both chaotic sensitivity and strong contraction, regardless of training.
We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs $\dot{h}=f_\theta(h)$ with saturating activations ($\tanh$, sigmoid, etc.): if $q$ hidden layers of the MLP $f_\theta$ satisfy $|\sigma'|\le\delta$ on a region~$U$, the input Jacobian is attenuated as $\norm{Df_\theta(x)}\le C(U)$ (for activations with $\sup_{x}|\sigma'(x)|\le 1$, e.g.\ $\tanh$ and sigmoid, this reduces to $C_W\delta^q$), forcing every Floquet (Lyapunov) exponen along any $T$-periodic orbit $\gamma\subset U$ into the interval $[-C(U),\;C(U)]$. This is a collapse of the Floquet spectrum: as saturation deepens ($\delta\to 0$), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with $\sigma'>0$, a saturation-weighted spectral factorisation yields a refined bound $\widetilde{C}(U)\le C(U)$ whose improvement is amplified exponentially in~$T$ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of $\tanh$-NODEs on the Morris--Lecar neuron model.
