Search papers, labs, and topics across Lattice.
The paper introduces EVGeoQA, a new benchmark for evaluating LLMs' ability to perform dynamic, multi-objective geo-spatial exploration, using electric vehicle charging scenarios with real-time user locations and dual objectives (charging and co-located activity). They also propose GeoRover, a tool-augmented agent framework for systematic evaluation. Experiments show that while LLMs can use tools for sub-tasks, they struggle with long-range spatial reasoning, but exhibit an emergent ability to summarize past trajectories for improved efficiency.
LLMs struggle with long-range spatial reasoning in dynamic environments, even when equipped with tools, highlighting a critical gap in their ability to perform real-world planning tasks.
While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial environments remains under-investigated. Existing Geo-Spatial Question Answering (GSQA) benchmarks predominantly focus on static retrieval, failing to capture the complexity of real-world planning that involves dynamic user locations and compound constraints. To bridge this gap, we introduce EVGeoQA, a novel benchmark built upon Electric Vehicle (EV) charging scenarios that features a distinct location-anchored and dual-objective design. Specifically, each query in EVGeoQA is explicitly bound to a user's real-time coordinate and integrates the dual objectives of a charging necessity and a co-located activity preference. To systematically assess models in such complex settings, we further propose GeoRover, a general evaluation framework based on a tool-augmented agent architecture to evaluate the LLMs' capacity for dynamic, multi-objective exploration. Our experiments reveal that while LLMs successfully utilize tools to address sub-tasks, they struggle with long-range spatial exploration. Notably, we observe an emergent capability: LLMs can summarize historical exploration trajectories to enhance exploration efficiency. These findings establish EVGeoQA as a challenging testbed for future geo-spatial intelligence. The dataset and prompts are available at https://github.com/Hapluckyy/EVGeoQA/.
