Search papers, labs, and topics across Lattice.
This paper introduces a neurosymbolic agent that integrates task-directed visual exploration with constrained symbolic planning to ensure the executability of plans generated for long-horizon household tasks. By leveraging a vision-language model to gather relevant predicates and a PDDL transition model for action constraints, the agent achieves over 90% success in executing plans in both VirtualHome and ALFWorld environments. Notably, the combination of constraints and search methods outperforms individual approaches, demonstrating a significant improvement in task completion rates while using fewer resources.
Combining visual exploration with symbolic planning enables agents to achieve over 90% success in executing complex household tasks, far surpassing traditional methods.
Language and vision-language models generate plausible embodied plans but do not guarantee executability, as their outputs can violate environment dynamics or act on incorrectly grounded entities. We present a neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning. In the first phase, a vision-language model and exploration harness acquire goal-relevant predicates and instance bindings from egocentric observations and grounded interactions, producing a symbolic initial state. In the second, a PDDL transition model restricts decoding to tokens that extend applicable actions. Monte Carlo tree search then evaluates executable continuations using a domain-independent planning heuristic. The resulting plans are executable by construction under the transition model, with transfer to the environment conditioned on correct visual grounding. On VirtualHome and ALFWorld, open 4B-27B models exceed 90% success in both environments, and our smallest agent substantially outperforms a 27B direct visual policy in each. Constraints and search prove complementary rather than interchangeable: in ALFWorld either alone solves under a third of tasks, whereas their combination solves over 95%. The method also uses several times fewer generated tokens than extended thinking and far fewer model-visible images than direct interaction, and residual failures localize to state acquisition rather than plan generation without any specialized training.
