Search papers, labs, and topics across Lattice.
This paper characterizes weakly regularized continual classification in homogeneous deep networks as sequential projections onto task margin sets, extending previous analyses limited to stationary models or continual linear frameworks. The authors reveal that while global convergence is often unattainable, local linear convergence can be guaranteed through specific regularity properties of these networks when subjected to random and cyclic task sequences. Additionally, they unify the analysis by extending their findings to continual regression, providing a comprehensive framework for homogeneous models.
Local linear convergence in homogeneous deep networks can be achieved even when global convergence fails, reshaping our understanding of continual learning dynamics.
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.
