Search papers, labs, and topics across Lattice.
This paper introduces Visually-grounded Humanoid Agents, a two-layer framework for creating autonomous digital humans that can perceive, reason, and act in novel 3D environments. The framework couples a World Layer that reconstructs semantically rich 3D Gaussian scenes from video with an Agent Layer that equips animatable avatars with first-person perception and embodied planning capabilities. Experiments demonstrate the agents' ability to perform goal-directed tasks with higher success rates and fewer collisions compared to baselines, showcasing a step towards scalable digital human population.
Imagine populating any 3D environment with digital humans that spontaneously navigate and interact, driven only by visual input and goals.
Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively animated, relying on privileged state or scripted control, which limits scalability to novel environments. We instead ask: how can digital humans actively behave using only visual observations and specified goals in novel scenes? Achieving this would enable populating any 3D environments with digital humans at scale that exhibit spontaneous, natural, goal-directed behaviors. To this end, we introduce Visually-grounded Humanoid Agents, a coupled two-layer (world-agent) paradigm that replicates humans at multiple levels: they look, perceive, reason, and behave like real people in real-world 3D scenes. The World Layer reconstructs semantically rich 3D Gaussian scenes from real-world videos via an occlusion-aware pipeline and accommodates animatable Gaussian-based human avatars. The Agent Layer transforms these avatars into autonomous humanoid agents, equipping them with first-person RGB-D perception and enabling them to perform accurate, embodied planning with spatial awareness and iterative reasoning, which is then executed at the low level as full-body actions to drive their behaviors in the scene. We further introduce a benchmark to evaluate humanoid-scene interaction in diverse reconstructed environments. Experiments show our agents achieve robust autonomous behavior, yielding higher task success rates and fewer collisions than ablations and state-of-the-art planning methods. This work enables active digital human population and advances human-centric embodied AI. Data, code, and models will be open-sourced.
CMU ML