Search papers, labs, and topics across Lattice.
The paper introduces Agentic Real2Sim, a novel framework that automates the conversion of real-world object-robot interactions into simulatable episodic twins, addressing the labor-intensive nature of traditional real-to-sim processes. By leveraging vision-language agents, the framework efficiently recovers scene geometries and object states while integrating various components into a cohesive physical simulation. Evaluations demonstrate that Agentic Real2Sim achieves comparable conversion success rates to existing methods at a significantly reduced cost, paving the way for scalable applications in robotics policy learning and evaluation.
Automating real-to-sim conversion with vision-language agents could revolutionize how we simulate robotic interactions, making it faster and cheaper than ever before.
Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on manual tuning of visual foundation models, mesh cleanup, coordinate-frame alignment, and brittle workflow glue across visual perception tools and simulators. We introduce \textit{Agentic Real2Sim}, a framework for generalized physical world modeling with vision-language agents, converting a real-world recording of object-robot interaction into a simulatable episodic twin which preserves observations, geometries, robot interactions, and object states. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines, marking a first step toward scalable conversion. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining comparable conversion success rate. We aim to use the resulting real-world-aligned twins for downstream robotics tasks, specifically policy learning and evaluation. The project site is available at https://agentic-real2sim.github.io/.
NUS