Search papers, labs, and topics across Lattice.
The paper introduces SympFormer, a novel accelerated attention block architecture inspired by inertial Nesterov-type dynamics on density spaces, where tokens carry both spatial and velocity variables. By discretizing and approximating accelerated density dynamics, they derive Hamiltonian momentum attention blocks that provably preserve elliptically contoured probability distributions in the linear self-attention setting. Empirical results demonstrate that SympFormer converges faster than classical attention blocks without increasing the number of oracle calls.
SympFormer achieves faster convergence in attention blocks by drawing inspiration from inertial Nesterov acceleration, offering a potential speedup without additional computational cost.
Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein-$2$-type metrics. We extend this viewpoint by introducing accelerated attention blocks derived from inertial Nesterov-type dynamics on density spaces. In our proposed architecture, tokens carry both spatial (feature) and velocity variables. The time discretization and the approximation of accelerated density dynamics yield Hamiltonian momentum attention blocks, which constitute the proposed accelerated attention architectures. In particular, for linear self-attention, we show that the attention blocks approximate a Stein variational gradient flow, using a bilinear kernel, of a potential energy. In this setting, we prove that elliptically contoured probability distributions are preserved by the accelerated attention blocks. We present implementable particle-based algorithms and demonstrate that the proposed accelerated attention blocks converge faster than the classical attention blocks while preserving the number of oracle calls.
