Search papers, labs, and topics across Lattice.
The paper introduces LEGS-POMDP, a novel POMDP system that integrates language, gesture, and visual observations to enable robots to search for objects in partially observable, open-world environments. LEGS-POMDP explicitly models uncertainty in both the target object's identity and its spatial location, addressing a key limitation of prior POMDP-based approaches. Experimental results in simulation and on a quadruped robot demonstrate that multimodal fusion significantly improves object search success rates compared to unimodal baselines, achieving 89% success in simulation.
Robots can now robustly search for objects in the real world by combining language, gestures, and vision within a POMDP framework that explicitly models uncertainty about object identity and location.
To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89\% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.
