Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for continual learning using recurrent fast-weight memories and selective state-space models, which compress expanding contexts into fixed-size states while maintaining causal relationships. By deriving normalized first-order updates for various regression objectives, the authors present several variants of their approach, including Falcon-1, Falcon-2, and Falcon-3, which demonstrate competitive performance in language modeling and improved length extrapolation in tasks like variable-digit addition. The framework effectively disentangles key aspects of recurrent sequence models, such as temporal alignment and forgetting, offering insights into optimizing online learning rules.
Fast-weight attention can enhance language modeling and improve task performance by effectively managing context and memory in recurrent networks.
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(蠒(k_{t-1}),v_t). The common same-step association (蠒(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
Tsinghua AI