Search papers, labs, and topics across Lattice.
The paper introduces RoboCurate, a synthetic data generation framework for robot learning that addresses the issue of inconsistent action quality in videos generated by video generative models. RoboCurate evaluates and filters the quality of annotated actions by comparing them with simulation replay, measuring the consistency of motion between the simulator rollout and the generated video. Experiments demonstrate that training on RoboCurate-generated data leads to significant improvements in robot learning performance across various tasks, including tabletop manipulation, pre-training for dexterous manipulation, and real-world humanoid dexterous manipulation.
By validating generated actions against physics simulation, RoboCurate unlocks a new level of realism in synthetic robot training data, leading to massive performance gains on real-world manipulation tasks.
Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video transfer to further augment appearance. We observe RoboCurate's generated data yield substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting.
