Search papers, labs, and topics across Lattice.
This paper investigates the impact of fine-tuning regimes, specifically the trainable parameter subspace, on continual learning (CL) performance. They formalize adaptation regimes as projected optimization over fixed trainable subspaces and hypothesize that method comparisons are not invariant across regimes. Through experiments with various CL methods (EWC, LwF, SI, GEM) and datasets, they demonstrate that the relative ranking of methods varies significantly depending on the trainable depth.
The best continual learning method for your task might depend more on *how much* of the model you fine-tune than *which* regularization strategy you use.
Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge. Despite substantial progress in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. In this paper, we argue that the fine-tuning regime, defined by the trainable parameter subspace, is itself a key evaluation variable. We formalize adaptation regimes as projected optimization over fixed trainable subspaces, showing that changing the trainable depth alters the effective update signal through which both current task fitting and knowledge preservation operate. This analysis motivates the hypothesis that method comparisons need not be invariant across regimes. We test this hypothesis in task incremental CL, five trainable depth regimes, and four standard methods: online EWC, LwF, SI, and GEM. Across five benchmark datasets, namely MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100, and across 11 task orders per dataset, we find that the relative ranking of methods is not consistently preserved across regimes. We further show that deeper adaptation regimes are associated with larger update magnitudes, higher forgetting, and a stronger relationship between the two. These results show that comparative conclusions in CL can depend strongly on the chosen fine-tuning regime, motivating regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.
