Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
F4R introduces an autonomous real-to-sim-to-real self-improvement loop that diagnoses real-world manipulation failures and reconstructs them into interactive, object-centric tabletop simulations. By pairing these failure-conditioned simulation twins with targeted reinforcement learning and sim-real co-training, the framework continually refines vision-language-action policies without the safety risks and bottlenecks of manual teleoperation. Across four physical manipulation tasks, the system attained 93.75% in-distribution and 90.0% out-of-distribution success, beating budget-matched behavioral cloning on real-world interventions by 18.75% OOD.
Real-world robot failures can be autonomously converted into interactive digital twins for targeted RL, boosting out-of-distribution manipulation by 18.75% while completely bypassing manual teleoperated recovery.
The real-world performance of current vision-language-action models is fundamentally constrained by the limited coverage of expert demonstrations and their insufficient understanding of physical interactions. A common remedy is to collect additional real-world demonstrations of newly encountered failures. However, this process is costly, inefficient, potentially unsafe, and difficult to scale. To address this challenge, we propose Failure for Rising (F4R), a failure-driven real-to-sim-to-real closed-loop learning framework that converts real-world failures into targeted policy improvement. F4R first uses an agent to automatically identify and diagnose failures from rollouts. It reconstructs each failure as an interactive, object-centric table-top environment that preserves the task-relevant spatial and physical conditions. The policy is then refined through failure-conditioned sim-real co-training followed by targeted reinforcement learning in the reconstructed environments. The improved policy is subsequently redeployed, while newly observed failures are continuously fed back into the next reconstruction and learning cycle. Real-world evaluations on four manipulation tasks show that F4R achieves 93.75% In-Distribution and 90.0% Out-of-Distribution (OOD) success, outperforming the budget-matched Targeted BC baseline by 18.75 percentage points under OOD conditions without collecting additional real-world corrective demonstrations.