Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
FlowAct-R2 couples a streaming multimodal reference diffusion transformer with a proactive agent to enable continuous, interactive humanoid video generation. To solve the accumulated motion drift and reactive limitations of standard talking avatars, the system introduces video-driven rotary positional embeddings, partially noised historical motion conditioning, and a decoupled pre-online/online planning architecture. The resulting system achieves stable, interruption-capable 720p generation in real time over hour-scale streaming sessions.
Humanoid video generation breaks out of short, reactive lip-sync loops into proactive, hour-scale 720p streaming that seamlessly handles live interruptions and dynamic reference shifts.
We present FlowAct-R2, a framework for interactive humanoid video generation that combines continuous multimodal control with proactive agent planning. Our method consists of two coupled components. First, a Streaming Multimodal Reference Diffusion Transformer adapts the pretrained Seedance 2.0 Mini reference-to-video backbone to accept rolling action prompts, streaming audio, and dynamically updated image, audio, and video references. Video-driven rotary positional embeddings align reference chunks with the generation timeline, while reference-plus-image conditioning and partially noised historical motion frames preserve appearance and avoid accumulated drift. Second, a Proactive Interaction Agent separates pre-online planning from online scheduling and response: it prepares a persona, a long-horizon agenda, and reusable multimodal skills in advance, then autonomously schedules behaviors, responds to audience input, and handles interruptions during a live session. FlowAct-R2 supports real-time 720p generation and hour-scale streaming across entertainment streaming, live shopping, video chatting, and live vlogging.