Search papers, labs, and topics across Lattice.
This paper introduces FlashMorph, a novel method for selecting layers in hybrid attention models that optimizes the balance between full and linear attention layers in Transformers. By formulating layer selection as a budget-constrained subset optimization problem, FlashMorph effectively identifies which layers should retain full attention while ensuring overall model efficiency. The results indicate that FlashMorph not only enhances long-context recall and general performance but also significantly reduces the computational cost associated with layer selection compared to traditional methods.
FlashMorph reveals that optimizing layer selection in hybrid attention models can drastically improve efficiency while maintaining performance, outperforming existing heuristic methods.
Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention. However, the effectiveness of Transformer-to-hybrid conversion critically depends on which layers preserve full attention. Existing hybrid layer selection methods typically rely on heuristic strategies such as fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking the interdependent layer effect under a global hybrid configuration. In this work, we formulate hybrid layer selection as a budget-constrained subset optimization problem. We further propose FlashMorph (Fast LAyer Selection for Hybrid MORPHing), an effective, efficient and scalable layer selection method for Transformer-to-hybrid conversion. FlashMorph first constructs a morphable model by equipping each full-attention layer with a converted linear-attention branch. It then freezes all model weights and jointly optimizes layerwise gates on synthetic long-context retrieval data, with a linearization regularization that encourages the model to rely on linear attention for efficiency. The learned gates are discretized under a preset full-attention budget to instantiate the hybrid architecture, followed by standard logits distillation and long-context finetuning. Extensive experiments show that FlashMorph discovers more effective hybrid configurations, preserves strong long-context recall and general benchmark performance while substantially reducing layer selection cost compared with existing layer selection methods, demonstrating its effectiveness, efficiency, and scalability.
