Search papers, labs, and topics across Lattice.
This paper introduces MultiGlobeQA, a comprehensive multilingual benchmark designed to evaluate geospatial reasoning in large language models (LLMs) through 46,060 question-answer pairs across 14 spatial-function families and 15 answer formats. The benchmark highlights significant performance gaps in LLMs, particularly in tasks requiring grid indexing and shape computation, where they struggle despite having access to geographic knowledge. Notably, the study reveals that while retrieval and tool use improve performance, LLMs still fail to exceed two-thirds accuracy, indicating that computational limitations, rather than knowledge access, are the primary challenge, especially in low-income regions.
LLMs falter in geospatial reasoning, with performance plateauing below two-thirds accuracy even when provided with gold facts, revealing computation as the key bottleneck.
Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.
