Search papers, labs, and topics across Lattice.
This study investigates the effects of correlated weight initialization in deep residual networks, revealing that such initializations can continuously interpolate between independent and perfectly correlated scenarios. The authors confirm a conjecture regarding the behavior of these networks, demonstrating that a unique critical scaling leads to a solution governed by a Young differential equation influenced by Hermite processes. Their findings establish the correlation structure and Hermite rank of the initialization as significant hyperparameters that dictate the asymptotic behavior of deep networks, contrasting with the universal Brownian motion observed under traditional iid initializations.
Correlated weight initialization can fundamentally alter the asymptotic behavior of deep residual networks, revealing critical hyperparameters that influence performance.
We study the large-depth behavior of residual networks whose weights are correlated across layers at initialization. Our results confirm and extend a conjecture of Marion et al. [2025], according to which correlated initializations should interpolate continuously between the Brownian stochastic differential equation arising from independent initialization and the ordinary differential equation arising from perfectly correlated initialization. When the initialization is obtained from the application of a feature function to a stationary Gaussian sequence with regularly varying correlation, we prove that there exists a unique critical scaling such that the infinite-depth limit is the solution of a Young differential equation driven by a Hermite process. Hermite processes reduce to the fractional Brownian motion if the feature function generating the initialization has Hermite rank one, which is the case for the identity function, for example. We show that the critical scaling and asymptotic limit are uniquely determined by the decay of correlations together with the Hermite rank of the feature function. Consequently, the correlation structure and Hermite rank of the initialization represent meaningful hyperparameters in the asymptotic regime. By contrast, under finite-variance iid initialization, the asymptotic driver is universally Brownian up to normalization regardless of the choice of distribution. Our proofs rely on a collection of novel results establishing a robust stability theory for Young differential equations in Banach spaces.
