Search papers, labs, and topics across Lattice.
The paper introduces 360CityArena, a benchmark designed to assess the urban exploration capabilities of embodied agents within a highly photorealistic environment derived from 360-degree videos of Tokyo's Akihabara district. By incorporating 602 video segments and 175 human-crafted tasks across three categories鈥擡nvironment Understanding, Path Reasoning, and Spatial Reasoning鈥攖he benchmark addresses the shortcomings of existing outdoor benchmarks in both realism and complexity. Evaluation results reveal that even advanced models like Gemini 2.5 Flash significantly underperform compared to human capabilities, achieving only 17.1% accuracy versus 77.3% for humans, highlighting the ongoing challenges in city-scale navigation for AI agents.
Even the most advanced AI models struggle with urban navigation, achieving only 17.1% accuracy compared to human performance of 77.3%.
We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Existing outdoor benchmarks either lack sufficient photorealism or complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360-degree video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories: Environment Understanding, Path Reasoning, and Spatial Reasoning, covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level (human: 77.3% vs. Gemini 2.5 Flash: 17.1%), revealing substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.
