Search papers, labs, and topics across Lattice.
The paper introduces EchoAvatar, a real-time framework for generating continuous full-body avatar motion from streaming audio, handling both speech and music without domain labels. A unified streaming architecture is used to synthesize motion from incremental audio inputs, trained with a strategy that enforces strong audio dependency for generalization. Reinforcement learning is used to refine online generation quality, and a tool-call interface allows LLMs to inject semantic control.
Finally, voice agents can become interactive humanoid avatars: EchoAvatar generates coherent full-body motion from streaming audio, even switching between speech and music, all while being controllable by an LLM.
Real-time synthesis of high-fidelity 3D character motion from audio is a pivotal component for next-generation interactive avatars and virtual assistants. However, most existing approaches are limited to offline processing of complete audio sequences or are constrained to specific domains, rarely handling both speech and music effectively. In this paper, we introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency. Central to our approach is a unified streaming architecture capable of synthesizing continuous motion from incremental audio inputs. We employ a robust training strategy that enforces strong audio dependency, allowing the model to seamlessly generalize across conversational speech and rhythmic music without requiring explicit domain labels or mode switching. Additionally, we explored Reinforcement Learning to refine the quality of online generation. Furthermore, we bridge reactive animation with intent-driven behavior via a tool-call interface that allows upstream Large Language Models to inject explicit semantic control. By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars. Extensive experiments demonstrate that our method outperforms state-of-the-art realtime baselines in motion quality and synchronization while maintaining the flexibility required for live deployment. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/EchoAvatar-Page.
