Search papers, labs, and topics across Lattice.
The paper introduces Pix2Act, an imitation learning method that reformulates 3D manipulation tasks into a more manageable 2D prediction problem by generating continuous image-space keypoint trajectories. This approach addresses challenges such as out-of-frame trajectories and limited precision by aligning observations and actions in the same coordinate space, allowing for equivariant transformations. The method demonstrates improved generalization and performance across various simulated and real-world manipulation tasks, outperforming existing state-of-the-art baselines while maintaining robustness under camera perturbations.
Pix2Act transforms complex 3D manipulation into a simpler 2D prediction task, leading to significant performance gains and robustness against camera variations.
Representing manipulation actions as 2D trajectories in the camera plane provides a compact and interpretable basis for learning complex 3D manipulation policies. However, it also creates challenges from out-of-frame trajectories and limited precision. We propose Pix2Act, an imitation learning method that addresses these challenges by generating continuous image-space keypoint trajectories in each camera plane and losslessly recovering end-effector poses via triangulation. This reformulates high-dimensional 3D control as a simpler, more learnable 2D prediction problem. Crucially, it aligns observations and actions in the same coordinate space, enabling equivariant transformations to jointly rotate individual camera images together with their image-space actions. We analyze the symmetry properties of this augmentation and design a network architecture that can fuse multiple camera views while respecting their per-view rotations. As a result, Pix2Act implicitly enlarges the support of the data distribution and learns invariant action structures across transformations, yielding improved generalization and overall performance. Across diverse simulated and real-world manipulation tasks, Pix2Act outperforms state-of-the-art baselines and remains robust under camera perturbations.
Stanford HAI