Search papers, labs, and topics across Lattice.
This paper introduces a physics-aware optimization framework for estimating simulation-ready scenes from real-world observations, focusing on cluttered environments. The method jointly recovers object shapes and poses by leveraging a shape-differentiable contact model and enforcing physical constraints. By exploiting the structured sparsity of the augmented Lagrangian Hessian, the method achieves efficient computation that scales well with scene complexity, enabling robust reconstruction of physically valid scenes.
Unlock robust real-to-sim transfer in cluttered environments with a physics-aware optimization that jointly recovers object shapes and poses, scaling efficiently with scene complexity.
Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end real-to-sim scene estimation pipeline that integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses.
