Search papers, labs, and topics across Lattice.
This paper introduces a framework for modeling structured multi-axis data, where transformations along independent axes do not depend on their order. By defining a journey operator that combines per-axis transformations, the authors show that their approach can recover existing methods like Rotary Position Embedding (RoPE) while also allowing for content-adaptive positional biases. Experimental results indicate that this framework, implemented in the JoFormer model, enhances performance in various tasks across vision and language domains by leveraging these inductive biases.
Journey operators reveal that compositional independence across axes can lead to significant performance improvements in multi-modal tasks, challenging traditional positional encoding methods.
Many kinds of data have structure along one or more axes: words in a sentence, pixels in an image, nodes in a tree, frames in audio, or cells in a 3D volume. Along one axis, order matters:"the dog bit the man"is different from"the man bit the dog."Across independent axes, however, neither composition nor movement should depend on the order of axes: in an image, composing right then down should give the same result as composing down then right, and moving right then down should describe the same relative position as moving down then right. We develop a framework for modeling this kind of multi-axis structure. Each data item carries its content together with a small transformation for each axis. A path connecting two positions defines a journey; the journey operator is the product of per-axis transformations along that path, governing both how data composes along the path and how relative position is described. When the transformations are fixed, our framework recovers Rotary Position Embedding (RoPE) and its multi-dimensional variants. When they depend on the data, the model gains a content-adaptive positional inductive bias. We show exactly when these paths are well-defined: both composition and movement across axes are path-independent precisely when the axis transformations commute. We also prove that, under the stated toral-frame symmetry, cocycle, bilinearity, and norm-preservation assumptions, the resulting pairwise scoring rule must take the form of block-wise rotations, explaining why RoPE-like methods arise naturally. Finally, we use this theory to design JoFormer, a model for value aggregation, and relate it to attention and state-space models (SSMs). Initial experiments across vision, language, and length generalization suggest that these inductive biases can have observable consequences in practice.
