Search papers, labs, and topics across Lattice.
This paper introduces nD-RoPE, a novel generalization of Rotary Position Embedding (RoPE) that extends its application to arbitrary dimensions by treating positions and frequencies as coupled \(n\)-dimensional vectors. The proposed method overcomes the limitations of existing approaches that either apply independent rotations or mix frequencies empirically, resulting in improved cross-dimensional interactions and isotropy. Experimental results across various high-dimensional data types, including images and videos, show significant performance enhancements and better generalization capabilities.
nD-RoPE achieves superior performance in high-dimensional contexts by ensuring isotropy and directionally balanced responses, challenging the limitations of traditional position embeddings.
Rotary Position Embedding (RoPE) is widely adopted in Transformer models, yet its extension to high-dimensional domains lacks a unified theoretical formulation. Most existing approaches either apply rotations independently along each axis or empirically mix frequencies, which limits cross-dimensional interactions and yields direction-dependent representations. To address these limitations, we propose nD-RoPE, a decomposition-free generalization of RoPE to arbitrary dimensions. From a translation-invariant formulation in continuous Hilbert space, we derive a spectral condition for isotropy that requires treating positions and frequencies as coupled \(n\)-dimensional vectors. We instantiate this formulation with a multi-scale regular-simplex wave-vector design, which provides non-degenerate spatial coverage and a symmetric, directionally balanced second-order response. Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.
