Search papers, labs, and topics across Lattice.
The paper introduces SYMBOLIZER, a framework that combines Visual Language Models (VLMs) with classical planning for Task and Motion Planning (TAMP) without requiring handcrafted symbolic action models. VLMs ground lifted predicates from images to obtain symbolic states, which are then used for domain-independent heuristic search. Experiments on ProDG and ViPlan benchmarks demonstrate that SYMBOLIZER achieves state-of-the-art results, outperforming direct VLM-based planning and matching the performance of methods using VLM-derived heuristics.
Forget handcrafted action models: this VLM-powered planner uses only lifted predicates and domain-independent search to achieve state-of-the-art results in complex task and motion planning.
Traditional Task and Motion Planning (TAMP) systems depend on physics models for motion planning and discrete symbolic models for task planning. Although physics model are often available, symbolic models (consisting of symbolic state interpretation and action models) must be meticulously handcrafted or learned from labeled data. This process is both resource-intensive and constrains the solution to the specific domain, limiting scalability and adaptability. On the other hand, Visual Language Models (VLMs) show desirable zero-shot visual understanding (due to their extensive training on heterogeneous data), but still achieve limited planning capabilities. Therefore, integrating VLMs with classical planning for long-horizon reasoning in TAMP problems offers high potential. Recent works in this direction still lack generality and depend on handcrafted, task-specific solutions, e.g. describing all possible objects in advance, or using symbolic action models. We propose a framework that generalizes well to unseen problem instances. The method requires only lifted predicates describing relations among objects and uses VLMs to ground them from images to obtain the symbolic state. Planning is performed with domain-independent heuristic search using goal-count and width-based heuristics, without need for action models. Symbolic search over VLM-grounded state-space outperforms direct VLM-based planning and performs on par with approaches that use a VLM-derived heuristic. This shows that domain-independent search can effectively solve problems across domains with large combinatorial state spaces. We extensively evaluate on extensively evaluate our method and achieve state-of-the-art results on the ProDG and ViPlan benchmarks.
