Search papers, labs, and topics across Lattice.
This paper introduces SparseCtrl-HOI, a sparse temporal control framework for generating Human-Object Interaction (HOI) videos, addressing the challenges of high annotation costs and limited motion synthesis diversity in existing dense guidance methods. By utilizing a Time-Controlled Rotary Positional Embedding (TiRoPE) to anchor keyframes and a Motion Prior Injection Module leveraging Multimodal Large Language Models (MLLMs), the approach enables realistic video generation with fewer keyframes. Evaluations demonstrate that SparseCtrl-HOI significantly reduces annotation overhead while enhancing the quality of synthesized videos for live-streaming e-commerce applications.
SparseCtrl-HOI slashes annotation costs while generating high-quality HOI videos, revolutionizing the landscape of AI-driven e-commerce content creation.
Human-Object Interaction (HOI) video generation aims to synthesize realistic videos of humans manipulating diverse objects, serving as a promising avenue for AI-driven live streaming e-commerce. A primary obstacle in this domain lies in the complexity of modeling fine-grained physical dynamics and the intricate spatial-temporal coordination between human hands and objects. Existing approaches to this problem typically rely on dense temporal guidance, e.g., frame-wise hand-object pose sequences, to strictly control the interaction process. However, such dense guidance incurs high annotation costs and affects motion synthesis diversity. To overcome these limitations, we introduce SparseCtrl-HOI, a novel sparse temporal control framework for HOI video generation. It requires only a few keyframes that capture interaction states at designated timestamps. Specifically, we employ a Time-Controlled Rotary Positional Embedding (TiRoPE) mechanism to temporally anchor these keyframes while preserving their spatial integrity. Subsequently, to govern the dynamics across intermediate frames, we propose a Motion Prior Injection Module that leverages Multimodal Large Language Models (MLLMs) to extract high-level motion priors. This empowers the model to hallucinate logically and physically plausible transitions. Furthermore, we build SparseHOI-5K, a high-quality and richly annotated dataset for HOI video generation with sparse temporal control. Comprehensive evaluations confirm that our method substantially reduces annotation overhead while synthesizing superior live-streaming e-commerce videos. Both our code and dataset are publicly available at https://mpi-lab.github.io/SparseCtrl-HOI.
