Search papers, labs, and topics across Lattice.
This paper introduces AnnotateAnything, an automatic annotation framework that transforms raw 3D assets into manipulation-ready assets by generating structured manipulation labels through two complementary pipelines. The first pipeline employs vision-language reasoning to infer object semantics and interaction constraints, while the second utilizes a physics annotation pipeline for grounding these insights in the geometry and physical properties of the assets. Experimental results show that AnnotateAnything significantly enhances annotation and data-collection efficiency, achieving higher task success rates compared to existing methods, and supports various downstream robotic tasks.
Achieving superior annotation efficiency and task success rates, AnnotateAnything revolutionizes how 3D assets are prepared for robot manipulation.
Simulation enables scalable robot data collection, but raw 3D assets provide only geometry, lacking the semantic, interactive, and physical knowledge needed to specify where and how robots should act. In this work, we present AnnotateAnything, a general automatic annotation framework that converts passive 3D assets into manipulation-ready assets with structured, diverse, and executable manipulation labels. AnnotateAnything is built around two complementary pipelines. First, a unified visual-language annotation pipeline using vision-language reasoning to infer object semantics, interaction constraints, and 3D-grounded cues, providing human-prior guidance for identifying meaningful interaction regions. Second, a fully automatic and massively parallel physics annotation pipeline grounds these priors in each asset's geometry and physical constraints through candidate generation, geometry optimization and trajectory generation. This pipeline produces diverse and executable action annotations, including grasp poses, dexterous contacts, articulation waypoints, insertion directions, hanging affordances, and navigation targets. Using the generated annotations, we further build an asynchronous parallel simulation data-collection system across diverse objects, tasks, and robot embodiments. Experiments demonstrate that AnnotateAnything achieves superior annotation efficiency, data-collection efficiency, and task success rates over existing annotation and data-generation pipelines, while also supporting downstream tasks such as affordance detection, robotic VQA, and visual instruction finetuning. We provide project materials on the project page and plan to release the full code, annotations, and benchmark to facilitate future research. Videos, code, demo assets, and annotations are provided in supplementary materials Project page: https://tourmaline-caramel-169490.netlify.app.
