Search papers, labs, and topics across Lattice.
This paper introduces ViserDex, a sim-to-real framework for monocular RGB in-hand object reorientation using 3D Gaussian Splatting (3DGS) for photorealistic rendering and domain randomization. By applying physically consistent augmentations directly to 3D Gaussians, they generate randomized visual data for training a robust pose estimator. The system, trained with curriculum-based RL and teacher-student distillation, achieves robust reorientation of diverse objects on a physical multi-fingered hand under challenging lighting, outperforming methods using conventional rendering.
Domain randomization directly in the 3D Gaussian Splatting representation space unlocks surprisingly robust sim-to-real transfer for monocular RGB in-hand manipulation, even with consumer-grade hardware.
In-hand object reorientation requires precise estimation of the object pose to handle complex task dynamics. While RGB sensing offers rich semantic cues for pose tracking, existing solutions rely on multi-camera setups or costly ray tracing. We present a sim-to-real framework for monocular RGB in-hand reorientation that integrates 3D Gaussian Splatting (3DGS) to bridge the visual sim-to-real gap. Our key insight is performing domain randomization in the Gaussian representation space: by applying physically consistent, pre-rendering augmentations to 3D Gaussians, we generate photorealistic, randomized visual data for object pose estimation. The manipulation policy is trained using curriculum-based reinforcement learning with teacher-student distillation, enabling efficient learning of complex behaviors. Importantly, both perception and control models can be trained independently on consumer-grade hardware, eliminating the need for large compute clusters. Experiments show that the pose estimator trained with 3DGS data outperforms those trained using conventional rendering data in challenging visual environments. We validate the system on a physical multi-fingered hand equipped with an RGB camera, demonstrating robust reorientation of five diverse objects even under challenging lighting conditions. Our results highlight Gaussian splatting as a practical path for RGB-only dexterous manipulation. For videos of the hardware deployments and additional supplementary materials, please refer to the project website: https://rffr.leggedrobotics.com/works/viserdex/
