Search papers, labs, and topics across Lattice.
This paper introduces an affordance foundation model designed to enhance functionality understanding in robot manipulation by predicting both where and how interactions should occur based on a single RGB-D observation and a language task description. The model significantly outperforms existing methods in affordance segmentation, contact-point prediction, and 3D motion prediction, achieving improvements of +23.9% in mean gIoU and +12.7–61.3% in hit-rate over the best baseline. This advancement enables robots to adapt to diverse environments and tasks without the need for finetuning or task-specific heuristics, facilitating real-world deployment.
Achieving a 23.9% improvement in affordance segmentation accuracy, this model enables robots to seamlessly adapt to diverse real-world manipulation tasks without additional training.
Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present ourmodel, a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, ourmodel predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate ourmodel from three aspects: for affordance segmentation, ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3; for contact-point prediction, it predicts substantially more accurate points, with a 12.7--61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. ourmodel can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN
