Search papers, labs, and topics across Lattice.
This paper investigates the robustness of similarity-based positional encoding (simPE) under rotational perturbations, a critical aspect for applications like medical imaging where geometric variations are common. The authors establish that while simPE is generally not rotation-invariant, it demonstrates stability under mild Lipschitz conditions, providing explicit bounds for perturbations in Frobenius norm. Experimental results across multiple datasets reveal that simPE significantly outperforms standard learned positional encodings in accuracy and other performance metrics when subjected to rotations, particularly at small to moderate angles.
simPE not only shows theoretical stability under rotations but also outperforms traditional positional encodings in real-world applications, especially in the medical imaging domain.
Positional encoding is a fundamental component of Transformer architectures, as it injects information about the spatial or sequential arrangement of inputs. Among recent alternatives to standard absolute and sinusoidal encodings, similarity-based positional encoding (simPE) has emerged as a flexible framework for representing positional structure through pairwise relations. simPE was originally designed for medical imaging applications, where geometric robustness is especially relevant: small rotations naturally arise during image acquisition, induced by imaging instruments, patient positioning, or slight acquisition misalignments. Despite its empirical promise, the theoretical behavior of simPE under geometric perturbations has not been fully characterized. In this paper, we study the robustness of simPE with respect to rotations, combining formal theoretical analysis with experimental validation. We first show that simPE is generally not rotation-invariant. We then prove that, under mild Lipschitz assumptions on the elementary components, simPE is stable under rotational perturbations and derive explicit perturbation bounds in Frobenius norm. We validate these findings experimentally on four controlled datasets--a synthetic Arrow dataset, a synthetic Shapes dataset (four geometric shape categories), a synthetic Digits dataset, and a benchmark image classification dataset (FashionMNIST)--in which training and validation images are kept in a fixed canonical orientation while test images are subjected to increasing rotation angles. Across all datasets, simPE consistently outperforms standard learned positional encoding in terms of accuracy, F1 score, precision, and recall under rotation, particularly in the small-to-moderate angle regime, corroborating the theoretical stability guarantees.
