Search papers, labs, and topics across Lattice.
This paper introduces RayPE, a novel positional encoding method for video diffusion transformers that incorporates 6D Plucker coordinates to enhance the representation of 3D scene geometry. By leveraging the bilinear nature of the Plucker reciprocal product, RayPE modifies the self-attention mechanism to include geometry terms alongside content, significantly improving camera controllability and 3D consistency in generated videos. Experimental results demonstrate that RayPE enhances overall video quality while maintaining a minimal increase in model parameters, validating its effectiveness across diverse datasets.
Injecting 6D Plucker coordinates into video transformers can drastically improve 3D consistency and camera control in generated videos.
Modern video diffusion transformers position their tokens through RoPE on the (u,v,t) axes -- a description of the camera's sampling grid that says nothing about the 3D structure of the scene. We observe that the geometric relation between two camera rays is captured by the Plucker reciprocal product, which is bilinear in the two rays -- the same algebraic form as the dot product in Transformer attention. Building on this analogy, we propose RayPE, a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention, with a query/key flip arrangement under which the symmetric identity configuration coincides exactly with the reciprocal product. The injection is additive, the resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms -- all of which our experiments find individually necessary. To make the encoding stable across video data with heterogeneous camera-translation scales (SfM, deep SLAM, metric), we further decouple ray direction from moment magnitude, gate the encoding by a learned function of the log-magnitude, and apply RMSNorm to align it with the QKNorm-normalized content branch. The full module adds less than 0.1% parameters to a pretrained video DiT, is zero-initialized to start from the pretrained weights, and improves camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset training mixture.
