Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
We introduce Urbex, an agentic vision-language framework that reframes city-scale 3D grounding from candidate ranking to active spatial search within an interactive multi-view environment. By enabling a VLM agent to query landmarks, zoom, render oblique views, and commit bounding boxes, Urbex achieves state-of-the-art performance on CityRefer with candidate-free grounding and significantly improved efficiency. This approach demonstrates superior cross-scene robustness and speed compared to existing ranking-based methods.
An agent can perform city-scale 3D object grounding by actively exploring a multi-view environment, bypassing the need for precomputed proposals and achieving state-of-the-art results.
City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.
DAMO
Tsinghua AI