Search papers, labs, and topics across Lattice.
This paper introduces CMP (Cognitive Memory Primitive), a continual-learning architecture that utilizes sparse relational codes and a two-tier competitive memory system, enabling local updates without end-to-end backpropagation. The authors demonstrate that CMP significantly reduces catastrophic forgetting compared to a parameter-matched Transformer using Elastic Weight Consolidation (EWC) across a controlled domain-incremental language modeling task. Results show that CMP achieves stable forgetting behavior and lower backward transfer, highlighting its potential as a robust approach for continual learning in AI systems.
CMP's innovative architecture cuts catastrophic forgetting in continual learning by leveraging sparse representations and local updates, outperforming traditional Transformer models.
We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?sents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns through local updates without end-to-end backpropagation through its feature-generating system. We investigate whether combining sparse representations, local learning, and persistent memory can reduce catastrophic forgetting relative to conventional backpropagation-based continual?learning approaches. On a controlled domain-incremental byte-level language modeling protocol, CMP demonstrates substantially lower backward transfer than a parameter-matched Trans?former trained with online Elastic Weight Consolidation (EWC). Across a three-seed replicated 15-domain experiment, CMP exhibits stable forgetting behavior, while separate head-to-head comparisons and domain-order analyses show consistently lower forgetting than the evaluated Transformer baseline under the reported experimental settings. We report these findings alongside a substantial single-domain accuracy gap relative to the Transformer, a null result on a vision benchmark, and a documented failure to combine CMP with an independent accuracy-improving mechanism, reflecting our commitment to reporting both positive and negative outcomes. These results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.
