Search papers, labs, and topics across Lattice.
This paper introduces StreamHOI, a novel framework designed for real-time human-object interaction (HOI) video generation, addressing the limitations of existing offline methods. By optimizing historical memory organization and employing bias-guided memory-specialized training, StreamHOI achieves a balance between latency and interaction fidelity. The results show that StreamHOI can generate long-duration HOI videos at 17.6 FPS with a first-chunk latency of just 0.75 seconds, outperforming both long-video baselines and recent HOI generation techniques in terms of interaction plausibility and object fidelity.
StreamHOI achieves real-time HOI video generation with impressive interaction fidelity, hitting 17.6 FPS and just 0.75 seconds latency.
Existing human--object interaction (HOI) video generation methods are largely limited to offline short-video generation with complex driving conditions, making them unsuitable for real-time interactive applications. We present \emph{StreamHOI}, a low-latency streaming framework for long-duration HOI video generation. Instead of converting heavily conditioned HOI pipelines into streaming systems, we study how an image-to-video streaming generator should organize historical memory to preserve interactions under bounded latency. We find that the standard sink-local memory design faces a trade-off in streaming HOI generation, and different transformer blocks show different historical-memory preferences for HOI regions and surrounding regions. To match memory composition with block behavior, StreamHOI performs offline HOI-aware block profiling and applies bias-guided memory-specialized training to adapt the generator to block-specific memory layouts. We further introduce a memory distance scaling module to strengthen long-range access to early interaction states. Extensive comparisons with both long-video baselines and recent HOI generation methods demonstrate that StreamHOI achieves strong interaction plausibility, object fidelity, human quality and efficiency, reaching 17.6 FPS with 0.75s first-chunk latency.
