Jinwoong Kim

Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea dnddl9456@hanyang.ac.kr, psj3493@hanyang.ac.kr *Corresponding author Abstract Self-attention encoders such as Bidirectional Encoder Representations from Transformers (BERT) scale quadratically with sequence length, making long-context modeling expensive. Linear-time state-space models, such as Mamba, are efficient; however, they show limitations in modeling global interactions and can suffer from padding-induced state contamination. We propose MaBERT, a hybrid encoder that interleaves Transformer layers for global dependency modeling with Mamba layers for linear-time state updates. This design alternates global contextual integration with fast state accumulation, enabling efficient training and inference on long inputs. To stabilize variable-length batching, we introduce padding-safe masking, which blocks state propagation through padded positions, and mask-aware attention pooling, which aggregates information only from valid tokens. On GLUE, MaBERT achieves the best mean score on five of the eight tasks, with strong performance on the CoLA and sentence-pair inference tasks. When extending the context from 512 to 4,096 tokens, MaBERT reduces training time and inference latency by 2.