Search papers, labs, and topics across Lattice.
This paper introduces a new setting, Multi-Object Affordance Grounding under Intent-Driven Instructions, to address the challenge of robots selecting the correct object for a task when multiple objects afford the same function. To facilitate research in this area, the authors present CompassAD, a benchmark dataset of 3D scenes with confusing object pairs and implicit natural language instructions. They also propose CompassNet, a framework incorporating Instance-bounded Cross Injection and Bi-level Contrastive Refinement, which achieves state-of-the-art results on the new benchmark and demonstrates effective transfer to real-world robotic grasping.
Robots can now better understand implicit instructions in cluttered scenes, choosing the knife over the scissors when told to "cut the apple," thanks to a new benchmark and neural network architecture designed to handle functionally similar objects.
When told to"cut the apple,"a robot must choose the knife over nearby scissors, despite both objects affording the same cutting function. In real-world scenes, multiple objects may share identical affordances, yet only one is appropriate under the given task context. We call such cases confusing pairs. However, existing 3D affordance methods largely sidestep this challenge by evaluating isolated single objects, often with explicit category names provided in the query. We formalize Multi-Object Affordance Grounding under Intent-Driven Instructions, a new 3D affordance setting that requires predicting a per-point affordance mask on the correct object within a cluttered multi-object point cloud, conditioned on implicit natural language intent. To study this problem, we construct CompassAD, the first benchmark centered on implicit intent in confusable multi-object scenes. It comprises 30 confusing object pairs spanning 16 affordance types, 6,422 scenes, and 88K+ query-answer pairs. Furthermore, we propose CompassNet, a framework that incorporates two dedicated modules tailored to this task. Instance-bounded Cross Injection (ICI) constrains language-geometry alignment within object boundaries to prevent cross-object semantic leakage. Bi-level Contrastive Refinement (BCR) enforces discrimination at both geometric-group and point levels, sharpening distinctions between target and confusable surfaces. Extensive experiments demonstrate state-of-the-art results on both seen and unseen queries, and deployment on a robotic manipulator confirms effective transfer to real-world grasping in confusing multi-object scenes.
