Search papers, labs, and topics across Lattice.
This paper introduces Gen2Physics, an automated framework that enhances the physical realism of generated 3D meshes by decomposing them into their material components, making them suitable for interactive applications. By employing a fine-tuned Vision Transformer for material segmentation and a Vision-Language Model for contextual refinement, the framework achieves significant improvements in material segmentation accuracy and produces watertight sub-meshes. Experimental results show that Gen2Physics more than doubles the segmentation accuracy compared to previous methods while maintaining volumetric mass estimation accuracy.
Gen2Physics doubles the accuracy of material segmentation in generated 3D meshes, enabling them to be immediately usable in interactive simulations and gaming.
While state-of-the-art generative models produce high-fidelity 3D meshes, these outputs lack the physical properties required for interactive simulation, gaming, or robotics. We introduce Gen2Physics, a unified and automated framework that grounds generated meshes in physics by automatically decomposing them into their constituent material components. Unlike prior approaches, which focus on volumetric representations incompatible with standard physics engines, Gen2Physics operates directly on meshes to produce immediately simulation-ready assets. Our pipeline integrates a fine-tuned Vision Transformer for dense material segmentation, a robust 2D-to-3D consistency projection, and a Vision-Language Model (VLM) guided refinement that leverages contextual reasoning to assign physical properties and infer internal geometry (solid vs. hollow). By converting surface patches into volumes with distinct densities, our method enables physically plausible dynamic simulations. Experimental results on the ABO-500 and PartNet-Material benchmarks demonstrate that Gen2Physics more than doubles the material segmentation accuracy of prior physics-grounding pipelines (15.6 to 48.3 mIoU), while matching the mass-estimation accuracy of volumetric methods and being the only approach to output watertight per-material sub-meshes.
