Search papers, labs, and topics across Lattice.
The paper introduces IMAGIN-4D, a diffusion-based generator that enhances human-object interaction (HOI) generation by using a reference image to provide spatio-temporal conditioning. This method addresses the ambiguity in interaction generation by extracting supervised interaction-state tokens for spatial conditioning and frame-aware tokens for temporal conditioning, allowing for nuanced control over body poses and object interactions. Experimental results demonstrate that IMAGIN-4D significantly improves interaction control and maintains motion quality compared to existing baselines, showcasing its effectiveness in generating coherent and contextually accurate animations.
IMAGIN-4D enables unprecedented fine-grained control over human-object interactions by leveraging spatio-temporal image conditioning, outperforming traditional methods that rely on single-token representations.
Generating human-object interactions (HOI) is central to character animation, robotics, AR/VR, and embodied AI. Recent HOI generation methods synthesize motion from text, object geometry, and sparse waypoints, controlling action semantics and object trajectories. However, these signals underspecify interaction: the same prompt and trajectory can produce different grasps, approach directions, body poses, object poses, contacts, and body-object layouts. We address this ambiguity with a reference image as a visual specification of the desired interaction snapshot. However, a single global image representation conflates distinct cues and conditions all frames on identical visual evidence. We therefore introduce IMAGIN-4D, a diffusion-based HOI generator that decomposes image conditioning spatio-temporally. For spatial conditioning, IMAGIN-4D extracts supervised interaction-state tokens for body pose, object pose, body-object contact, and spatial relationships at the depicted frame. For temporal conditioning, it computes frame-aware tokens by querying image patches per generated frame, allowing sequence segments to attend to different visual cues from the same image. To balance image, text, and waypoint cues, IMAGIN-4D uses role-aware conditioning: text, waypoints, and interaction-state tokens use separate AdaLN streams, while frame-aware visual tokens cross-attend with motion tokens. Since HOI motion datasets lack paired images, we build a synthetic motion-to-image rendering pipeline from FullBodyManipulation (FBM) and introduce an image-adherence metric to evaluate whether generated motions match the reference snapshot. Experiments on FBM and BEHAVE show that IMAGIN-4D improves fine-grained interaction control over single-token and uniformly image-conditioned baselines while preserving waypoint-following and motion quality. Code and models will be released at https://imagin4d.github.io.
