Search papers, labs, and topics across Lattice.
This paper introduces CARVE, a novel recurrent model that addresses key limitations of the delta-rule architecture by implementing a content-aware erase mechanism that operates on the key axis rather than the value axis. This approach not only preserves memory efficiency but also enhances the model's performance, achieving state-of-the-art results on common-sense reasoning benchmarks and retrieval tasks with significantly lower resource usage compared to existing models. With a perplexity of 15.72 on WikiText and substantial improvements in memory and parameter efficiency, CARVE demonstrates the potential for more effective recurrent architectures in large-scale language modeling.
CARVE achieves state-of-the-art performance with 19% fewer parameters while maintaining competitive perplexity, reshaping expectations for recurrent model efficiency.
Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify. This memory-blind gating is one of three coupled defects in the leading delta-rule architecture (GDN-2): the value-axis erase mask wastes parameters at the scale of the value projection, and -- as we prove -- mathematically prevents the WY-form triangular chunk solver that makes recurrent training competitive with Transformers. We introduce CARVE (Content-Aware Recurrent with Value Efficiency), which resolves all three problems through one principle: erase only on the key axis. This is provably necessary and sufficient for the WY-form solver to remain valid. Within it, CARVE reuses the recurrent output tensor -- already written to GPU memory -- as a free content signal for the erase gate, and replaces the per-value write-gate projection with a single scalar per head. At initialisation CARVE is bit-identical to GDN-2; any quality difference emerges from what the content gate learns. At 1.3B parameters trained on 100B tokens, CARVE achieves WikiText perplexity 15.72 (minus 0.18 vs. GDN-2, a 4.5-sigma effect), leads every recurrent baseline on nine common-sense reasoning benchmarks, and sets state of the art on every RULER retrieval probe -- at 0.4% throughput overhead, 13% lower peak memory, and 19% fewer parameters. Six formal theorems cover memory capacity, Lyapunov stability, gradient flow, expressivity separation, Pareto-optimal chunk size, and hybrid optimality.
