Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a manipulation planning system that leverages affordance recognition and action effect prediction to generate action plans based on text-based goals. By employing a multi-modal goal-matching module, the system effectively tracks object positions even when occluded, allowing for robust planning in dynamic environments. The integration of a real-to-sim image conversion module enhances the system's ability to generalize across varied visual appearances, demonstrating effective manipulation planning in both simulated and real-world scenarios.
Achieving manipulation planning that remains robust even when objects are occluded or visually altered could revolutionize how robots interact with complex environments.
We present a manipulation planning system based on affordance recognition and action effect prediction. The system reasons through possible futures in visual form, and evaluates candidate plans by agreement of predicted outcomes with text-based goals set at run-time, using a multi-modal goal-matching module. Positions of objects named in the goal text are tracked through predictions even when occluded, making it possible to generate action plans even when objects become occluded, or when their initial descriptors cease to identify them in future states. We further expand the system with an image conversion module for translating real-world state images with objects of varied shapes and visual appearances into a consistent visual appearance, to facilitate manipulation planning in a physical robot setup. We evaluate performance of the system's modules in isolation and demonstrate the integrated system's manipulation planning capabilities on a set of challenging tasks in both simulation and on hardware.