Search papers, labs, and topics across Lattice.
PhyGenHOI generates physically plausible 4D human-object interactions from text prompts by coupling a motion diffusion model (MDM) for human motion with a material point method (MPM) simulation for object dynamics, both represented as 3D Gaussian Splats. They introduce a windowed attraction loss to synchronize motion, a contact-driven re-simulation step for momentum transfer, and a masked video-SDS objective to improve contact fidelity. Results demonstrate improved physical consistency and visual quality compared to existing methods across diverse actions, humans, and objects.
Synthesizing realistic human-object interactions is now possible by directly coupling generative human motion with explicit physics simulation, opening the door to more believable virtual environments.
We address the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Given a static 3D human and target object represented as 3D Gaussian Splats (3DGS), our goal is to synthesize dynamic scenes where the human actively engages with the object through actions, such as punching or kicking, in accordance with a given input text. To this end, we introduce PhyGenHOI, a novel framework that couples generative human motion with an explicit physical object simulation. We model the human as a semantic agent driven by a Motion Diffusion Model (MDM) and the object as a physical agent simulated via the Material Point Method (MPM), utilizing 3D Gaussians as a unified, differentiable representation. We supervise their interaction through three coupled mechanisms: (1) A Windowed Attraction Loss that temporally synchronizes generative motion to intercept the object; (2) A Contact-Driven Re-simulation step that triggers physically consistent momentum transfer upon impact; and (3) A Masked Video-SDS objective that injects video-based priors to enhance contact fidelity. Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines. Project page and videos: https://omerbenishu.github.io/PhyGenHOI/
