Search papers, labs, and topics across Lattice.
Hoi3DGen generates high-quality 3D human-object interactions from text by addressing limitations in existing score distillation approaches. The method leverages multimodal large language models to curate a realistic and high-quality interaction dataset. This curated data is then used to train a text-to-3D pipeline, resulting in significant improvements in interaction fidelity and 3D model quality compared to existing baselines.
Text-to-3D generation of human-object interactions jumps 4-15x in text consistency thanks to a new pipeline trained on data curated with multimodal LLMs.
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
