Search papers, labs, and topics across Lattice.
This paper presents a data augmentation framework that leverages real-world eye-in-hand demonstrations to generate visually realistic fisheye image sequences and corresponding action trajectories for robotic manipulation. By employing a novel Gaussian Splatting formulation tailored for wide field-of-view fisheye cameras, the authors enhance the robustness of visuomotor policies against out-of-distribution observations caused by minor changes in initial configurations or unseen obstacles. Experimental results demonstrate that this approach significantly improves success rates in manipulation tasks, both in familiar and augmented environments with collision avoidance challenges.
Action-view augmentation can transform how robots adapt to unforeseen obstacles, boosting manipulation success rates significantly.
Visuomotor policies for manipulation have demonstrated remarkable potential in modeling complex robotic behaviors, yet minor alterations in the robot's initial configuration and unseen obstacles easily lead to out-of-distribution observations. Without extensive data collection effort, these result in catastrophic execution failures. In this work, we introduce an effective data augmentation framework that generates visually realistic fisheye image sequences and corresponding physically feasible action trajectories from real-world eye-in-hand demonstrations, captured with a portable parallel gripper with a single fisheye camera. We introduce a novel Gaussian Splatting formulation, adapted to wide FoV fisheye cameras, to reconstruct and edit the 3D scene with unseen objects. We utilize trajectory optimization to generate smooth, collision-free, view-rendering-friendly action trajectories and render visual observations from corresponding novel views. Comprehensive experiments in simulation and the real world show that our augmentation framework improves the success rate for various manipulation tasks in both the same scene and the augmented scene with obstacles requiring collision avoidance.
Stanford HAI