Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Parallel Hybrid Architecture (PHA), which integrates Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs) as independent parallel branches, each optimized for specific tasks in long-context modeling. By allowing these components to specialize—GSS for global context, attention for selective retrieval, and FFN for additional processing—PHA achieves significant improvements in efficiency and perplexity. On benchmarks like WikiText-103 and OpenWebText, PHA outperforms existing models while offering up to 40% lower memory usage and 24% higher throughput, showcasing a promising approach to long-range dependency challenges in NLP.
Separating sequence modeling into specialized parallel branches allows for Transformer-level perplexity with dramatically improved efficiency in long-context tasks.
Modeling long-range dependencies remains a central challenge in natural language processing. Transformer architectures achieve strong performance via self-attention but scale quadratically ($O(N^2)$) with sequence length, while State Space Models (SSMs) scale linearly ($O(N)$) but suffer from a selective recall bottleneck, struggling to retrieve precise information from compressed states. This creates a fundamental tradeoff between efficiency and perplexity. To tackle these challenges, we propose the \textit{Parallel Hybrid Architecture (PHA)}, which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs) as independent parallel branches fused by a learnable mixing mechanism. Instead of forcing SSMs to approximate attention or serializing the two paradigms, PHA allows each branch to specialize: GSS captures global context, while attention performs selective retrieval, with FFN providing complementary processing. On WikiText-103, PHA achieves 16.51 PPL at 125M parameters, outperforming Hedgehog (16.70) and H3-125M (23.70). Scaling to 180M parameters yields 16.42 PPL, which gives comparable results with the pure attention baseline while delivering 24\% higher throughput and up to 40\% lower memory usage at long contexts. On OpenWebText, our 125M model achieves 19.72 PPL, outperforming standard Transformers (20.60) and GSS hybrid baselines (19.80). These results demonstrate that separating sequence modeling paradigms into parallel specialists enables Transformer-level perplexity with substantially improved efficiency for long-context language modeling.