Search papers, labs, and topics across Lattice.
This study investigates how modular architecture, task similarity, and representational dimensionality influence compositional continual learning in a sequential A-B-A paradigm. By comparing a task-partitioned recurrent network to a single-network baseline across high- and low-dimensional regimes, the authors find that while both architectures perform similarly in high-dimensional settings, modularity significantly enhances performance in lower-dimensional contexts by creating task-specific subspaces that adaptively overlap or separate based on task similarity. These results highlight the importance of representational dimensionality in determining the functional benefits of modular structures in continual learning scenarios.
Modularity in neural architectures can dramatically improve compositional learning, but only in lower-dimensional settings where task representations are rich and nuanced.
Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference. We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning in a sequential A-B-A paradigm, comparing a task-partitioned recurrent network to a single-network baseline while inducing high- and low-dimensional regimes via weight-scale manipulations. In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry, suggesting that explicit modular structure has little impact when representations are weakly constrained. In a lower-dimensional "rich" regime, modularity becomes decisive: the modular network develops graded task-specific subspaces that overlap for similar tasks, partially align for moderately dissimilar tasks, and separate for dissimilar tasks, yielding a more compositional and interpretable organization than the single network. These findings identify the representational regime induced by initialization scale, which co-varies with representational dimensionality, as a key factor governing when compositional, modular structure is functionally beneficial in continual learning, and support viewing safety and robustness as problems of adaptive allocation of representational subspaces rather than fixed separation versus sharing.
