Search papers, labs, and topics across Lattice.
GeoAgentBench (GABench) is introduced as a dynamic benchmark for evaluating tool-augmented LLM agents in Geographic Information Systems (GIS), addressing the limitations of existing static benchmarks. The benchmark integrates 117 GIS tools across 53 tasks and 6 domains, using a Parameter Execution Accuracy (PEA) metric and VLM-based verification to assess parameter inference and spatial accuracy. A novel "Plan-and-React" agent architecture is proposed, which decouples global planning from reactive execution, demonstrating improved performance over traditional frameworks in multi-step reasoning and error recovery across seven LLMs.
LLMs struggle with GIS tasks due to parameter misalignment and runtime anomalies, but a "Plan-and-React" architecture that mimics expert cognitive workflows significantly improves performance.
The integration of Large Language Models (LLMs) into Geographic Information Systems (GIS) marks a paradigm shift toward autonomous spatial analysis. However, evaluating these LLM-based agents remains challenging due to the complex, multi-step nature of geospatial workflows. Existing benchmarks primarily rely on static text or code matching, neglecting dynamic runtime feedback and the multimodal nature of spatial outputs. To address this gap, we introduce GeoAgentBench (GABench), a dynamic and interactive evaluation benchmark tailored for tool-augmented GIS agents. GABench provides a realistic execution sandbox integrating 117 atomic GIS tools, encompassing 53 typical spatial analysis tasks across 6 core GIS domains. Recognizing that precise parameter configuration is the primary determinant of execution success in dynamic GIS environments, we designed the Parameter Execution Accuracy (PEA) metric, which utilizes a "Last-Attempt Alignment" strategy to quantify the fidelity of implicit parameter inference. Complementing this, a Vision-Language Model (VLM) based verification is proposed to assess data-spatial accuracy and cartographic style adherence. Furthermore, to address the frequent task failures caused by parameter misalignments and runtime anomalies, we developed a novel agent architecture, Plan-and-React, that mimics expert cognitive workflows by decoupling global orchestration from step-wise reactive execution. Extensive experiments with seven representative LLMs demonstrate that the Plan-and-React paradigm significantly outperforms traditional frameworks, achieving the optimal balance between logical rigor and execution robustness, particularly in multi-step reasoning and error recovery. Our findings highlight current capability boundaries and establish a robust standard for assessing and advancing the next generation of autonomous GeoAI.
