Search papers, labs, and topics across Lattice.
This paper introduces Variational-Ising-Attention (VIA), a novel attention mechanism that enhances traditional softmax normalization by incorporating an Ising model to capture interactions between entities. The motivation stems from the need for richer context modeling in scientific tasks, where long-token constraints are not a limiting factor. Experimental results on retrosynthesis reaction center prediction show that VIA significantly outperforms standard softmax attention, highlighting the importance of tailored attention in scientific applications.
Tailored attention mechanisms can outperform traditional softmax approaches by leveraging structured interactions, as demonstrated by VIA's success in predicting retrosynthesis reaction centers.
Attention enables context modeling via query-key scoring with softmax normalization. Driven by industrial long-context demands, mainstream research has converged toward sparsity and efficiency--yet softmax's independence assumption persists. For scientific tasks unburdened by long-token constraints, however, richer structured coupling may often be essential, making tailored attention both viable and more appropriate. To this end, we propose Variational-Ising-Attention (VIA), which augments softmax normalization with an interacting Ising model; attention patterns emerge from learnable pairwise couplings via variational mean-field inference, redefining attention from a ranking over isolated items to a collective state over interacting entities. We instantiate VIA on retrosynthesis reaction center prediction, a task inherently governed by cooperative bond-breaking constraints. Comprehensive experiments across model variants, coupled with mechanistic analyses, demonstrate that VIA consistently and substantially outperforms standard softmax attention. More broadly, our findings suggest that for scientific problems, the optimal solution is not general-purpose efficiency, but appropriately tailored attention aligned with intrinsic domain structure. This work provides a theoretically grounded and empirically validated instantiation of this paradigm.
