Search papers, labs, and topics across Lattice.
This paper introduces RoPESim, a simulation framework for evaluating robot manipulation policies before real-world deployment by predicting potential failures. The framework uses a VLM-based scenario generator to create simulated environments, a diffusion model to simulate manipulation action trajectories, and an evaluator to assess collision, manipulation failure, and completion rate. Experiments across five scenarios demonstrate the framework's ability to generate at least one feasible action trajectory per scenario, validated through real robot execution.
Mitigate real-world robot manipulation failures by simulating and evaluating action trajectories *before* deployment, using a novel VLM- and diffusion-based framework.
Predicting the robot manipulation plan prior to real-world execution is an important capability for robots to complete tasks in manufacturing environments. However, current AI-based manipulation planning methods lack this capability, making it difficult to deploy them in real-world manufacturing scenarios. In this work, we propose a simulation-based human-robot collaboration framework to evaluate predicted robot actions before real-world execution. The framework consists of a VLM-based scenario generator, a diffusion-based action simulator, and an evaluator. First, the scenario generator automatically creates a simulation scenario with objects and obstacles identified and placed. Then, the action simulator generates a series of manipulation action trajectories using a diffusion model in the simulation environment. Each action trajectory is assessed by the evaluator for collision failure, manipulation failure, and completion rate. The final evaluation results are returned to the user for verification and approval. In our experiment, we apply our framework to five chosen scenarios with highly potential collision failures. For each scenario, at least one feasible planned action trajectory is generated. It is then verified through real robot execution, demonstrating the effectiveness of the proposed framework.
