Search papers, labs, and topics across Lattice.
MaBERT, a hybrid encoder, interleaves Transformer layers for global dependency modeling with Mamba layers for linear time state updates, achieving efficient long context modeling. Padding-safe masking and mask-aware attention pooling are introduced to stabilize variable length batching. Experiments show MaBERT achieves strong GLUE performance and reduces training/inference time by over 2x compared to baselines when extending context to 4096 tokens.
By interleaving Transformers and Mamba, MaBERT achieves a 2x speedup in training and inference for long-context masked language modeling, without sacrificing GLUE benchmark performance.
Self attention encoders such as Bidirectional Encoder Representations from Transformers(BERT) scale quadratically with sequence length, making long context modeling expensive. Linear time state space models, such as Mamba, are efficient; however, they show limitations in modeling global interactions and can suffer from padding induced state contamination. We propose MaBERT, a hybrid encoder that interleaves Transformer layers for global dependency modeling with Mamba layers for linear time state updates. This design alternates global contextual integration with fast state accumulation, enabling efficient training and inference on long inputs. To stabilize variable length batching, we introduce paddingsafe masking, which blocks state propagation through padded positions, and mask aware attention pooling, which aggregates information only from valid tokens. On GLUE, MaBERT achieves the best mean score on five of the eight tasks, with strong performance on the CoLA and sentence pair inference tasks. When extending the context from 512 to 4,096 tokens, MaBERT reduces training time and inference latency by 2.36x and 2.43x, respectively, relative to the average of encoder baselines, demonstrating a practical long context efficient encoder.
