Search papers, labs, and topics across Lattice.
The paper introduces Thalamically Routed Cortical Columns (TRC$^2$), a novel decoder-only architecture designed to improve continual learning in language models by mitigating catastrophic forgetting under non-stationary data distributions. TRC$^2$ combines sparse thalamic routing over cortical columns with mechanisms for modulation, prediction, memory, and feedback, alongside a fast corrective pathway for rapid adaptation. Empirical results demonstrate that TRC$^2$ improves the stability-plasticity tradeoff on language modeling and continual learning benchmarks, enabling rapid on-stream adaptation with minimal forgetting at comparable compute costs.
Language models can now learn continuously without forgetting, thanks to a brain-inspired architecture that selectively updates parameters.
Continual learning is a core requirement for deployed language models, yet standard training and fine-tuning pipelines remain brittle under non-stationary data. Online updates often induce catastrophic forgetting, while methods that improve stability frequently increase latency, memory footprint, or dense computation in ways that do not scale well to long contexts. We introduce TRC$^{2}$ (Thalamically Routed Cortical Columns), a decoder-only backbone that addresses continual learning at the architectural level. TRC$^{2}$ combines sparse thalamic routing over cortical columns with mechanisms for modulation, prediction, memory, and feedback, together with a fast corrective pathway that supports rapid adaptation without destabilizing slower parameters. The resulting block is sparse and chunk-parallel, enabling efficient training and inference while preserving clean ablations of each subsystem. We instantiate a reproducible training and evaluation stack and a continual-learning harness that measures proxy forgetting under streaming domain shifts. Across language modeling and continual learning benchmarks, TRC$^{2}$ improves the stability-plasticity tradeoff at comparable compute, enabling rapid on-stream adaptation while preserving previously acquired behavior.
