Search papers, labs, and topics across Lattice.
EgoPhys is a novel framework that constructs deformable physical digital twins from egocentric RGB video, addressing the challenge of predicting complex dynamics of elastic materials and fabrics. By distilling per-object inverse-physics solutions into a compact codebook, EgoPhys enables the prediction of dense spring stiffness fields for unseen objects without the need for test-time optimization. The framework outperforms existing methods in reconstruction, future prediction, and zero-shot generalization, and is validated on a real robotic platform, showcasing its potential for practical applications in robotics and computer vision.
EgoPhys can predict the physics of unseen deformable objects from just a single egocentric video, revolutionizing how we approach digital twin generation in robotics.
Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from egocentric RGB-only video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an egocentric interaction dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6 robot, demonstrating that a digital twin initialized from a single egocentric human play video can serve as an internal world representation to aid in deformable-object planning, highlighting egocentric RGB observations as a scalable path toward real-to-sim pipelines.
