Search papers, labs, and topics across Lattice.
This paper introduces Dual Attention Residuals (DAR), a novel architecture that enhances Transformer models by enabling reciprocal cross-stream addressing between multi-stream residual pathways and historical retrieval. The method effectively computes depth weights from one stream to influence the information retrieval of another, leading to improved validation loss across various model sizes, including dense and sparse architectures. Key findings reveal that DAR not only outperforms standard residual Transformers but also maintains depth-wise diversity, avoiding issues of redundancy and functional imbalance seen in other multi-stream designs.
Reciprocal cross-stream addressing in Dual Attention Residuals leads to significant improvements in Transformer performance without sacrificing depth-wise diversity.
Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.
