Search papers, labs, and topics across Lattice.
The paper introduces Vid2Sid, a video-driven system identification pipeline that uses foundation-model perception and a VLM-in-the-loop optimizer to calibrate robot simulator physics parameters by analyzing paired sim-real videos. This approach diagnoses mismatches, proposes parameter updates with natural language rationales, and addresses the challenges of limited sensing and perception noise in sim-to-real transfer. Experiments on a tendon-actuated finger and a deformable tentacle demonstrate that Vid2Sid achieves comparable or superior performance to black-box optimizers on sim2real holdout controls, while also providing interpretable reasoning, and recovers ground-truth parameters more accurately in sim2sim validation.
Forget hand-tuning your robot simulator: this system uses paired real-world videos and language models to automatically diagnose and fix physics mismatches with interpretable rationales.
Calibrating a robot simulator's physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. Sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13\% vs. 28--98\%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.
