Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to cloth manipulation by integrating a simulator-in-the-loop optimization framework that leverages a physical simulator for real-time trajectory evaluation and refinement. The method employs a synthetic data generation pipeline and a real-to-sim module that translates RGB observations into simulation-compatible states, enhancing the robot's ability to manipulate cloth effectively. Experimental results demonstrate that this approach significantly improves success rates and robustness in real-world cloth manipulation tasks compared to existing baseline methods.
Real-time cloth manipulation success rates soar as robots learn to refine actions using a simulator-in-the-loop approach, outperforming traditional methods.
Simulator-in-the-loop optimization offers a promising inference-time mechanism for robot manipulation. It uses a physical simulator as a backend rollout engine to evaluate candidate trajectories in parallel and refine nominal actions online, a paradigm proven effective in rigid-body manipulation where state and contact are relatively tractable. We bring this paradigm to real-world cloth manipulation from a single RGB input through three pillars. (i) We design a scalable synthetic-data generation and inference-time rollout pipeline built on FLASH, a deformable-object simulator that provides a practical balance among physical fidelity, numerical stability, and rollout efficiency. (ii) We develop a real-to-sim module, trained purely on synthetic data, that maps a single RGB observation to simulation-compatible cloth state by fusing pretrained visual features with learnable canonical tokens. (iii) We perform online planning by coupling a sparse-mesh rollout backend with prior-guided MPPI, anchored at an offline-distilled policy trajectory, preserving manipulation-relevant deformation and contact while enabling sufficient parallel rollout batches. Real-robot experiments show higher success rates and stronger robustness than baseline methods.
NUS