Search papers, labs, and topics across Lattice.
The paper introduces SARAH, a real-time, causal method for generating spatially-aware conversational human motion for embodied agents in VR, addressing the limitations of current methods that lack spatial awareness beyond speech-aligned gestures. SARAH combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio to produce full-body motion that aligns gestures with speech while orienting the agent towards the user. The method incorporates a gaze scoring mechanism with classifier-free guidance, enabling users to control eye contact intensity at inference time, and achieves state-of-the-art motion quality at over 300 FPS on the Embody 3D dataset.
Finally, VR agents can maintain natural gaze and respond to user movement in real-time, thanks to a novel architecture that achieves state-of-the-art motion quality at over 300 FPS.
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.
CMU ML