Search papers, labs, and topics across Lattice.
This paper introduces AdaPlanBench, a novel benchmark designed to evaluate the adaptive planning capabilities of Large Language Model (LLM) agents under progressively revealed world and user constraints. By utilizing a dynamic multi-turn interaction protocol across 307 household tasks, the study reveals that even the best-performing LLMs achieve only 67.75% accuracy when faced with dual constraints, with performance notably declining as the number of constraints increases. The findings underscore the significant challenges LLMs face in reliably adapting to dynamically disclosed constraints, particularly highlighting the impact of user constraints on planning effectiveness.
LLMs struggle with adaptive planning, achieving only 67.75% accuracy when faced with progressively revealed world and user constraints.
Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing benchmarks still underexplore adaptive planning under such progressively revealed dual constraints. To address this gap, we introduce AdaPlanBench, a dynamic interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints. At runtime, agents interact with the environment in a multi-turn protocol where hidden constraints are revealed only when the agent proposes a plan that violates them, requiring iterative plan revision under accumulating feedback. This makes planning challenging, as agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM agents.
