Search papers, labs, and topics across Lattice.
Lngram v2 enhances the memory capacity of transformers by decoupling the memory dimension from the backbone width and introducing a context-aware grouped-query attention mechanism. This approach allows for significant reductions in both total and activated memory parameters while maintaining or improving performance in language modeling tasks across various vision-language models. Notably, Lngram v2 enables the semantic recovery of continuous hidden states from discrete IDs, offering a structured interface for analyzing internal model representations.
Lngram v2 achieves substantial memory efficiency gains while preserving semantic integrity, allowing for scalable transformer architectures without the typical parameter bloat.
Transformers lack a native lookup mechanism, requiring repeated dense computation to recognize and reuse local static patterns. Lngram v1 introduces tokenizer-independent conditional memory through discrete latent n-gram addressing, but its memory capacity is coupled with the backbone width, limiting scalability due to high parameter and activation costs. We propose Lngram v2, which decouples the number of routes, memory dimension, and backbone width, and introduces a context-aware grouped-query attention readout to scale memory capacity independently. A zero-value Sink and counterfactual surrogate gradients further improve readout selectivity and routing trainability while preserving hard discrete addressing. Experiments across vision--language models (VLMs) of different scales show consistent improvements, including successful scaling to a 30B-parameter model. Compared with Lngram v1, Lngram v2 substantially reduces both total and activated memory parameters while maintaining or improving language modeling performance. Further analysis shows that its discrete IDs preserve substantial semantic structure of continuous hidden states, enabling semantic recovery from IDs alone and stable ID--semantic associations across datasets. These results establish Lngram v2 as an efficient and scalable latent conditional memory mechanism whose discrete addresses also provide a structured interface for analyzing internal model representations.
