Search papers, labs, and topics across Lattice.
This paper introduces Orthogonal Hyper-Connections (oHC) as a novel approach to stabilize training in Transformer architectures by constraining the residual mixing matrix to the rotation group SO(4) via quaternions. By preventing amplification or attenuation of residual streams, oHC maintains diversity across layers, addressing the limitations of previous Hyper-Connection models that risked destabilizing training. Experimental results demonstrate that oHC outperforms traditional single-stream residuals and other Hyper-Connection methods across various downstream tasks, highlighting its effectiveness in enhancing model performance.
Training stability in Transformers can be achieved by constraining residual connections to orthogonal transformations, preventing the loss of diversity across layers.
Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n \times n$ residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We prove that inside this set the mixing step can reduce the norm of the residual streams only by shrinking the differences between the streams, while their mean is left unchanged; and since the reduction accumulates over layers, the streams grow more alike and their diversity is spent with depth. We therefore propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction, which keeps training stable and no longer forces the differences between the streams to contract. Specifically, at the four streams used by recent HC models we parameterize the group in closed form by a pair of unit quaternions, which adds no parameters, replaces the iterative projection with a fixed pattern of signed additions, and can be constructed faster than mHC. We evaluate oHC across a comprehensive set of downstream tasks, where it outperforms the single-stream residual baseline, mHC and iHC, which fixes the residual matrix to the identity.
