Search papers, labs, and topics across Lattice.
This paper introduces Mean Root Square Normalization (MRSNorm), a novel approach that addresses the limitations of traditional Root Mean Square Normalization by preserving phase manifold properties and ensuring gradient stability. By pairing channels into 2D phasors and employing a global $L_1$ average, MRSNorm effectively reduces the number of learnable parameters by half while maintaining structural integrity during optimization. Empirical results demonstrate that MRSNorm outperforms standard normalization techniques in terms of stability, particularly under extreme hyperparameter settings, suggesting a significant shift in deep representation learning methodologies.
Halving parameters while ensuring gradient stability, MRSNorm redefines normalization by preserving phase properties and preventing numerical explosion in deep networks.
While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.
