Search papers, labs, and topics across Lattice.
This paper introduces GeoProp, a novel adapter that aligns proprioceptive data with visual inputs through geometric grounding and spatial feature sampling, addressing the limitations of traditional fusion methods. By projecting the robot's state onto the image plane and using FiLM modulation to inject spatial priors into visual features, GeoProp enhances the robot's ability to ground its state within the scene. The approach significantly boosts performance across 67 tasks, achieving an average improvement of 10.6% in real-world scenarios while maintaining a minimal increase in parameter count.
GeoProp achieves a remarkable 10.6% boost in real-world manipulation tasks by effectively grounding robot state in visual context, all while adding minimal complexity.
Proprioception is fundamental to robotic manipulation, yet standard fusion methods often treat it as an isolated vector lacking explicit alignment with visual tokens. Without a direct correspondence between 3D kinematics and 2D feature maps, manipulation policies struggle to ground the robot's state within the scene, frequently underperforming even vision-only baselines. To address this, we introduce GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp projects the robot state onto the image plane to sample localized visual features, constructing a grounded state token. It then injects state-derived spatial priors into the corresponding visual features via FiLM modulation. To capture motion intent, GeoProp further samples features at a short-horizon predicted coordinate derived from recent kinematics, providing look-ahead visual context. Across 67 tasks, GeoProp improves Diffusion Policy by 8.7% on 63 simulation tasks and pi_0 by 4.0% on the RoboTwin subset, and yields a 10.6% average gain across both policy families in the real world, while adding only 2-3% to the parameter count. These results demonstrate that GeoProp is a simple yet high-impact inductive bias for generalist embodied policies. Project page: https://alibaba-damo-academy.github.io/GeoProp/.
DAMO