Search papers, labs, and topics across Lattice.
The paper introduces MixFormer, a linear Transformer that enhances long-context modeling by integrating a Mixture-of-Memory-Experts (MoE) mechanism to address the limitations of existing State Space Models (SSMs). By employing a Time-Aware Linear Attention (TALA) mechanism, MixFormer dynamically updates memory through learnable exponential decay functions, allowing it to selectively reinforce crucial historical information and reduce memory dilution. Experimental results on long-sequence text and image generation tasks show that MixFormer significantly outperforms existing models while offering a more sustainable computational framework for future web infrastructure.
MixFormer achieves substantial performance improvements in long-context tasks by dynamically managing memory with a novel Mixture-of-Memory-Experts approach.
State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.
