Search papers, labs, and topics across Lattice.
The paper introduces Gauge-Invariant Spectral Transformers (GIST), a novel graph transformer architecture designed to address the computational and gauge invariance challenges in adapting transformers to graph-structured data. GIST achieves $\mathcal{O}(N)$ complexity via random projections and preserves gauge invariance through inner-product-based attention, enabling discretization-invariant learning and parameter transfer across mesh resolutions. Empirical results demonstrate that GIST matches state-of-the-art performance on graph benchmarks and achieves state-of-the-art aerodynamic prediction on large-scale mesh-based Neural Operator benchmarks.
You can now train graph transformers that generalize across different mesh resolutions, thanks to a new architecture that maintains gauge invariance while scaling linearly.
Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end $\mathcal{O}(N)$ complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets.
