Search papers, labs, and topics across Lattice.
This paper addresses the challenges in evaluating and comparing affordance prediction methods due to inconsistent formulations and reporting practices. By introducing the Affordance Sheet, a comprehensive documentation framework, the authors aim to standardize task formulations, input modalities, and experimental protocols, thereby enhancing transparency in benchmarking. The key result is that the Affordance Sheet facilitates reproducible evaluations and reliable comparisons of affordance models, which is crucial for their application in real-world scenarios and ensuring human safety.
The Affordance Sheet could revolutionize how we benchmark and evaluate multimodal affordance prediction models, ensuring reproducibility and safety in real-world applications.
Affordance prediction is the identification of potential actions an agent can perform on a target object from multimodal inputs. Affordance prediction methods are difficult to evaluate and compare due to heterogeneous problem formulations, inconsistent dataset annotations, incomplete reporting of experimental protocols, and limited information about deployment conditions. These limitations challenge fair benchmarking and performance comparison. To promote transparency, we propose the Affordance Sheet, a documentation detailing task formulation with its input modalities, model architectures and training information, datasets, and experimental protocols. Affordance Sheets enable reproducible benchmarking and reliable evaluation of affordance models for real-world scenarios, including generalisation to novel conditions and human safety.
