Search papers, labs, and topics across Lattice.
This study introduces Urban Planning Bench (UPBench), a novel evaluation framework designed to assess the reasoning capabilities of large language models (LLMs) in urban planning contexts. By evaluating 25 LLMs across four knowledge pillars and five cognitive levels, the research reveals a non-monotonic cognitive curve where models excel at higher-order analytical tasks but struggle with context-sensitive and regulatory aspects of planning. The findings highlight significant limitations in LLMs' ability to replicate the nuanced judgment required in professional urban planning, suggesting that while they can assist in certain tasks, they cannot fully replace human expertise in critical areas.
LLMs outperform expectations in analytical reasoning but falter on context-sensitive tasks essential for urban planning, revealing a critical gap in AI's applicability in this field.
Problem, Research Strategy, and Findings: The rise of large language models (LLMs) raises a key question for urban planning: which forms of professional planning knowledge can AI replicate, and which still require human judgment? Although AI tools are increasingly used in planning practice, there is still no systematic framework for testing whether they can reason with the contextual sensitivity, value awareness, and institutional literacy central to planning expertise. This paper introduces Urban Planning Bench (UPBench), a domain-specific evaluation framework that assesses LLM reasoning through a 4x5 matrix of four knowledge pillars and five cognitive levels adapted from Bloom's revised taxonomy. Evaluating 25 LLMs with automated scoring and expert review, we find a non-monotonic cognitive curve: models perform better on higher-order analytical tasks than on factual recall and integrative judgment. This suggests that planning knowledge often treated as lower-order is deeply shaped by institutional, jurisdictional, and temporal context, making it hard for LLMs to generalize. We summarize these limits as four epistemic diagnostics: regulatory hallucination, conceptual conflation, wickedness paralysis, and phronetic deficit. Takeaway for Practice: The findings support differential delegation in planning. LLMs can assist with cross-disciplinary synthesis, literature review, scenario generation, and preliminary policy analysis. However, they remain unreliable for jurisdiction-specific regulation, normative conflict resolution, and context-sensitive procedure. Agencies should require verification for AI-assisted regulatory analysis, while planning education should emphasize institutional literacy, normative judgment, and contextual sensitivity.
