Search papers, labs, and topics across Lattice.
This paper introduces InteractiveAvatar, a novel framework for real-time streaming video generation that addresses the challenges of visual temporal consistency and user intent perception in avatar interactions. By employing autoregressive distillation and a Long-Short Visual Memory (LSVM) mechanism, the framework maintains coherence over extended durations while effectively compressing historical visual data. The integration of a Reasoning-Reaction Module (RRM) allows avatars to respond accurately to user intent, achieving state-of-the-art performance in both visual consistency and interactive capabilities across diverse scenarios.
Achieving real-time avatar generation that maintains visual consistency and responds intelligently to user intent could revolutionize interactive streaming experiences.
Recent diffusion-based models have enabled realistic audio-driven avatar generation in real-time streaming. However, existing approaches struggle to maintain visual temporal consistency and fail to explicitly perceive user intent in complex interactive streaming scenarios. To address these challenges, we propose InteractiveAvatar, a real-time infinite-streaming video generation framework that supports visually consistent avatar video generation and intent-aware interactions. With autoregressive distillation, InteractiveAvatar achieves real-time str-eaming generation of human avatars over arbitrarily long durations. For visual consistency, we introduce a Long-Short Visual Memory (LSVM) mechanism that flexibly compresses historical visual information into compact tokens, preserving both short-range coherence and long-term consistency. To generate avatars with speeches and actions aligned with user intent, we propose a Reasoning-Reaction Module (RRM), which incorporates a State-Cycling strategy and a Cache-Switching mechanism. Extensive experimental results over diverse scenarios demonstrate that our method achieves state-of-the-art visual consistency in long-duration generation, while enabling complex user-avatar interaction in real time.
