Search papers, labs, and topics across Lattice.
GaussTwin introduces a real-time digital twin framework for robotic manipulation that unifies physically grounded simulation using position-based dynamics and Cosserat rods with efficient rendering and visual correction via Gaussian splatting. The system anchors Gaussians to physical primitives and enforces SE(3) updates based on photometric error and segmentation masks to achieve stable prediction-correction while maintaining physical fidelity. Experiments on a Franka Research 3 platform demonstrate improved tracking accuracy, robustness, and the ability to perform downstream tasks like push-based planning compared to baselines.
By fusing Gaussian splatting with physically-based simulation, GaussTwin offers a digital twin that's both visually accurate and physically plausible, unlocking new possibilities for robot learning and control.
Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.
