Search papers, labs, and topics across Lattice.
This study investigates the uneven performance of large language models (LLMs) on planning tasks, proposing that these discrepancies stem from two distinct planning competencies rather than a single ability spectrum. By employing a multidimensional item response theory model on the ACPBench-Hard dataset, the authors identify operational reasoning and structural enumeration as the two principal dimensions influencing planning performance. The results indicate that while operational reasoning benefits from model scaling and longer reasoning traces, structural enumeration shows relative insensitivity, prompting a shift towards competency-level evaluations in LLM planning assessments.
Unearthing two distinct planning competencies in LLMs reveals that scaling up models enhances operational reasoning but leaves structural enumeration largely unchanged.
When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.
