Search papers, labs, and topics across Lattice.
LaGSplat introduces a novel framework that infers physics-governed dynamics from monocular video, enabling real-time interaction with both rigid and deformable objects. By leveraging a low-dimensional latent state that serves dual roles in a learned Lagrangian and a Gaussian Splatting decoder, the system allows users to apply unseen external forces and observe realistic object responses. Validation across various test cases demonstrates LaGSplat's capability to handle complex scenarios, outperforming traditional pixel-space and neural-field decoders in maintaining plausible dynamics under unseen conditions.
Real-time interaction with physics-governed simulations from monocular video is now possible, even with unseen forces.
We present LaGSplat (Latent Lagrangian Gaussian Splatting), a framework that infers interactive, physics-governed dynamics from one or a few monocular videos. At inference it lets a user push on the filmed object, rigid or deformable, with an external force that was never measured, annotated, or seen during training. This is possible because a low-dimensional latent state $\mathbf{q} \in \mathbb{R}^d$ plays two roles at once: it is the generalised coordinate of a learned dissipative Lagrangian and the conditioning variable of a Gaussian Splatting decoder. The inductive bias of this decoder, whose primitives are explicit points $渭_i(\mathbf{q})$ that move with the object, is what lets a force $f$ applied in the image pull back into a latent generalised force $J(\mathbf{q})^\top f$ and enter the equations of motion, which pixel-space (CNN) or neural-field (NeRF) decoders cannot do. We validate LaGSplat on test cases of increasing difficulty, from rigid to deformable and from autonomous to forced real systems, combining monocular video and sensor measurements. We further demonstrate interactive use: forces of arbitrary magnitude and direction can be applied to the reconstructed object at any time, its response rendered in real time, in 2D or 3D. Assuming a dissipative Euler-Lagrange equation over a few generalised coordinates trades generality for a bounded, plausible response to unseen forces, where an unconstrained predictor diverges.
