Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of open-weight large language models (LLMs) in repairing Planning Domain Definition Language (PDDL) models by leveraging both positive and negative test plans. The research reveals that while LLMs can achieve a notable F1 score improvement to 0.87 compared to a symbolic baseline of 0.49, the overall reliability in satisfying test constraints remains inadequate, particularly in complex domains. The findings highlight a significant gap in the ability of LLMs to ensure robust automated model repair, with pass rates dropping dramatically in challenging scenarios.
LLMs can significantly outperform traditional symbolic methods in PDDL model repair, but their reliability still falters in complex domains.
AI planning is concerned with finding a sequence of actions that achieves a specified goal. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language (PDDL). An active line of research investigates how errors in such models can be detected and repaired. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution. Automated repair methods then modify the PDDL model to satisfy these constraints. In this paper, we evaluate the ability of recent open-weight large language models to perform this repair task using an LLM-only approach. Our experiments show that the symbolic baseline achieves an $F_1$ score of $.49$, while the best-performing LLM reaches $.87$ with high reasoning effort, an absolute improvement of $.38$. However, that setting has a mean test pass rate of only $.82$, falling to $.06$ on the Thoughtful domain; even the best setting that includes the test traces reaches only $.92$. Thus, current open-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair.
