Search papers, labs, and topics across Lattice.
This paper introduces Retrieval-Augmented Robotics (RAR), a paradigm where robots actively retrieve and utilize visual procedural manuals from unstructured corpora to perform tasks in zero-shot settings. The approach formulates task execution as an iterative Retrieve-Reason-Act loop, enabling robots to ground abstract 2D diagrams to 3D physical parts via cross-modal alignment and synthesize executable plans. Experiments on a long-horizon assembly benchmark demonstrate that RAR significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval.
Robots can now assemble complex furniture in zero-shot settings by actively retrieving and reasoning over visual instructions, outperforming methods relying solely on internal knowledge or limited examples.
To achieve general-purpose utility, we argue that robots must evolve from passive executors into active Information Retrieval users. In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the robot or embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval. This work establishes the basis of RAR, extending the scope of Information Retrieval from answering user queries to driving embodied physical actions.
