Search papers, labs, and topics across Lattice.
This paper introduces EgoAfford, a benchmark that integrates task-oriented affordance grounding with egocentric observations and multi-step planning for complex tabletop tasks. By utilizing approximately 15.5k human-verified images and a multimodal large language model called EgoLens, the authors demonstrate how to effectively segment functional regions of objects involved in actions while generating action plans. The results indicate that EgoLens achieves strong performance on both generated and real-world observations, highlighting the intricate relationship between perception and planning in robotic tasks.
EgoAfford reveals that effective task-oriented affordance grounding can significantly enhance multi-step planning in complex environments.
Part-level affordance grounding has advanced the localization of functional object regions associated with elemental actions. Extending this capability to complex tasks calls for connecting the semantic roles of participating objects with task-state-aligned visual observations and multi-step planning. We introduce EgoAfford, a benchmark designed to connect these three aspects. Given an egocentric observation and a high-level tabletop task, a model must generate the remaining plan and segment the functional regions of up to three components of the next action: the direct object, instrument, and destination. EgoAfford comprises approximately 15.5k human-verified images from 2,000 generated multi-step scenes, organized as semantically aligned, task-complete image series, together with EgoAfford-Real, 102 manually captured images spanning 26 tasks. We further present EgoLens, a 3B multimodal large language model with role-specific mask decoders, as an in-domain reference model for this joint task. Evaluations of recent referring-segmentation MLLMs, commercial-VLM--SAM2 pipelines, and EgoLens highlight the complementary challenges of next-step inference and action-role-conditioned part grounding. EgoLens establishes strong reference performance on both generated and manually captured observations. Together, EgoAfford and EgoLens provide a foundation for jointly studying perception and planning in multi-step tabletop tasks. Our project page is available at: https://egoafford.github.io
