Search papers, labs, and topics across Lattice.
This paper introduces the gate-zero growth operator, a novel function-preserving mechanism for continual learning that integrates new residual blocks via a zero-initialized gate. By ensuring rank separation in the functional Jacobian, the method effectively minimizes old-domain forgetting while allowing for controlled functional variation as new gates are activated. Experimental results demonstrate that gate-zero growth achieves near-zero forgetting on a large Transformer model, significantly outperforming traditional methods in preserving old knowledge during adaptation to new tasks.
Near-zero forgetting in continual learning is achievable with a geometric approach that preserves old knowledge while integrating new information seamlessly.
We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank separation} in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is $O(\|\boldsymbol{\alpha}\|^2)$ and Jacobian leakage $O(\|\boldsymbol{\alpha}\|_\infty)$, giving a controlled departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting ($\Delta_A<0.1$) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control ($G_{\text{stack}}$) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
