Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the ability of Vision-Language Models (VLMs) to infer semantic affordances for non-humanoid robots, a key challenge for deploying VLMs in diverse robotic applications. They introduce a hybrid dataset combining real-world robot data with VLM-generated synthetic data to evaluate VLM performance across various robot morphologies and object categories. The key finding is that VLMs exhibit inconsistent affordance inference capabilities, characterized by low false positive rates but high false negative rates, especially for novel tool use.
VLMs are surprisingly conservative when inferring affordances for non-humanoid robots, exhibiting a high rate of missed opportunities for interaction, particularly with novel tools.
Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding human-object interactions, but their application to robotic systems with non-humanoid morphologies remains largely unexplored. This work investigates whether VLMs can effectively infer affordances for robots with fundamentally different embodiments than humans, addressing a critical gap in the deployment of these models for diverse robotic applications. We introduce a novel hybrid dataset that combines annotated real-world robotic affordance-object relations with VLM-generated synthetic scenarios, and perform an empirical analysis of VLM performance across multiple object categories and robot morphologies, revealing significant variations in affordance inference capabilities. Our experiments demonstrate that while VLMs show promising generalisation to non-humanoid robot forms, their performance is notably inconsistent across different object domains. Critically, we identify a consistent pattern of low false positive rates but high false negative rates across all morphologies and object categories, indicating that VLMs tend toward conservative affordance predictions. Our analysis reveals that this pattern is particularly pronounced for novel tool use scenarios and unconventional object manipulations, suggesting that effective integration of VLMs in robotic systems requires complementary approaches to mitigate over-conservative behaviour while preserving the inherent safety benefits of low false positive rates.