Search papers, labs, and topics across Lattice.
The paper introduces GASE, an automated system that reconstructs high-fidelity simulation environments using multi-view video streams from panoramic camera arrays. By employing a camera-pose-based strategy for robust object extraction and high-quality scene inpainting, GASE significantly enhances segmentation accuracy and maintains a minimal performance gap in real-robot deployments. Experimental results show that GASE outperforms existing methods by over 10% in segmentation accuracy while achieving state-of-the-art inpainting quality, making it a vital tool for effective sim-to-real transitions in robot learning.
GASE reduces the sim-to-real performance gap to under 10% while outperforming existing methods in segmentation accuracy by over 10%.
Training embodied agents in the real world requires skilled operators and expensive hardware. Simulation environments offer a compelling alternative by enabling large-scale, cost-effective data augmentation. Consequently, rapidly constructing high-fidelity simulation scenes with a minimal sim-to-real gap has become a critical objective in robot learning. While reconstruction-based methods provide superior visual quality, current workflows are hindered by inefficient data acquisition and subpar foreground object extraction. We thus propose GASE, a highly automated system for simulation scene construction. GASE leverages multi-view video streams from panoramic camera arrays to enable rapid environment scanning. To ensure high-quality asset generation, our pipeline introduces a camera-pose-based strategy that robustly extracts objects across frames in the 2D domain, followed by high-fidelity scene inpainting. Foreground objects and the static background are then reconstructed independently and seamlessly imported into physics simulators for policy training. Extensive experiments demonstrate that GASE outperforms existing 3D Gaussian-based methods in segmentation accuracy by over 10\% while achieving state-of-the-art inpainting quality. Furthermore, real-robot deployments across manipulation and navigation tasks maintains a performance gap of less than 10\% compared to policies trained purely on real-world data. These results confirm that GASE provides an efficient and highly effective solution for bridging the sim-to-real gap. Code will be released.
