Search papers, labs, and topics across Lattice.
The authors investigate the fidelity gap in natural-language-to-PDDL translation by establishing an iterative generation-critic-repair pipeline that integrates syntax parsing, automated planner validation, and domain-conformance checks. Standard evaluation metrics like syntactic validity and operational solvability are fundamentally flawed, routinely masking severe model hallucinations in initial states, object mappings, and goal specifications. Across Planetarium, AutoPlanBench, and PDDL 2.1 benchmarks, the results reveal that operational planner success diverges drastically from ground-truth semantic reconstruction, though structured symbolic-plus-LLM feedback loops partially remediate these discrepancies.
Solvable does not mean faithful: automated planners routinely validate and execute LLM-generated PDDL instances that completely corrupt the user's intended initial states and goal specifications.
Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.
