Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome the rigid parameter sharing and monolithic retrieval of prior n-gram lookup memories like Engram, FactorEngram factorizes local pattern representations into sparse linear combinations over a shared basis dictionary. By scoring the backbone hidden state against these basis vectors, the architecture enables fine-grained, context-dependent gating of individual memory components rather than applying a single scalar gate to a monolithic embedding. Evaluated on 340M- and 1B-parameter Transformer backbones, this approach yields consistent gains across language modeling and downstream tasks, with architectural ablations pinpointing pre-attention placement in middle layers as optimal.
Monolithic n-gram memory tables waste capacity and fail on polysemous context, but decomposing lookups into a shared basis dictionary with per-component gating unlocks scalable, context-aware parameter expansion.
Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized n-gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token n-grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration.