Search papers, labs, and topics across Lattice.
This paper introduces Communicability-Inspired Positional Encoding (CIPE), a novel approach to positional encodings for Transformers that effectively addresses the challenges of non-Euclidean graphs. By leveraging communicability, which aggregates contributions from paths of all lengths, CIPE defines an Attention-Compatible Geometry that enhances the inner products used in self-attention mechanisms. Empirical results demonstrate that CIPE improves performance by an average of 35.5% across seven benchmarks, significantly outperforming existing positional encodings in both structure-agnostic and structure-biased contexts.
CIPE transforms how Transformers encode positional information in non-Euclidean graphs, achieving a remarkable 35.5% performance boost over traditional methods.
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging. Such encodings should ideally induce an Attention-Compatible Geometry for self-attention: not merely describing graph structure, but defining a geometry whose inner products reflect meaningful structural relatedness. To realize this geometry, we propose Communicability-Inspired Positional Encoding (CIPE), built from communicability, a measure between pairs of nodes that aggregates contributions from paths of all lengths. By construction, CIPE inner products recover communicability, converting global multi-path connectivity into an attention-ready similarity geometry. For practical Transformer training, we introduce dimensionality alignment, mapping graph-size-dependent CIPE representations to prescribed dimensions while faithfully preserving the induced geometry. Empirically, CIPE improves structure-agnostic Transformers by 35.5% on average across seven benchmarks, outperforming representative PEs; it also consistently improves structure-biased graph Transformers, where competing PEs often yield only marginal benefits. These results position CIPE as a principled framework for attention-compatible graph positional encodings.
