Search papers, labs, and topics across Lattice.
This paper analyzes signal propagation in finite-width linear recurrent models, deriving exact formulas for hidden state signal energies under complex Gaussian initialization. It identifies three distinct depth-width scaling regimes: subcritical ($t=o(\sqrt n)$) where infinite-width approximations hold, critical ($t\sim c\sqrt n$) where deviations emerge, and supercritical ($t\gg \sqrt n$) where finite-width effects dominate. The analysis reveals that finite-width effects accumulate faster in recurrent models compared to feedforward networks, impacting the stability of standard initialization schemes like Glorot.
Infinite-width approximations, a cornerstone of neural network theory, crumble much faster in recurrent models than previously thought, failing beyond a depth of order $\sqrt{n}$.
We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth $t$ grows jointly with width $n$. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large $t$. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth-width scaling regimes that govern signal propagation: (i) a subcritical regime $t=o(\sqrt n)$, in which the infinite-width approximation remains valid; (ii) a critical regime $t\sim c\sqrt n$, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime $t\gg \sqrt n$, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results demonstrate that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones, leading to qualitatively different signal propagation behavior.
