Search papers, labs, and topics across Lattice.
This paper introduces GROW$^2$, a novel approach for open-world affordance grounding that enables robots to creatively utilize tools by hierarchically splitting the grounding process into semantic and geometric levels. By leveraging Vision-Language Models for commonsense reasoning and vision foundation models for precise 3D localization from RGB-D images, GROW$^2$ effectively identifies suitable tools and their relevant parts for various tasks. Experimental results demonstrate that GROW$^2$ surpasses state-of-the-art methods in affordance prediction and achieves zero-shot generalization in both simulated and real-world scenarios, significantly enhancing robotic tool use capabilities.
GROW$^2$ enables robots to creatively use open-category tools with zero-shot generalization, outperforming existing methods in both simulated and real-world tasks.
Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of $\textit{open-world affordance grounding}$: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW$^2$ (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW$^2$ harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.
