Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a joint 2D-3D segmentation and association framework for street-level imagery, integrating visual semantics with multi-view geometric reasoning via structure-from-motion. The method replaces traditional 2D multi-object tracking with a 3D-driven association mechanism that leverages geometric consistency for identity preservation across wide-baseline viewpoints. Experiments show a 22% performance gain in challenging urban scenarios compared to state-of-the-art 2D-only tracking methods, demonstrating improved coverage and robust identity retention.
Ditch 2D tracking limitations: this new approach uses 3D geometry to boost object tracking in street-level imagery by 22%.
Accurate interpretation of street-level imagery is essential for large-scale urban mapping and the creation of Spatial Digital Twin (SDT) environments. This work presents a unified framework for joint 2D-3D segmentation and association that integrates visual semantics with multi-view geometric reasoning. Unlike conventional approaches that rely heavily on sequential frames for temporal tracking, our method leverages zero-shot detection and segmentation together with structure-from-motion reconstruction to establish stable cross-view correspondences. A 3D-driven association mechanism replaces traditional 2D multi-object tracking, using geometric consistency to guide identity preservation across wide-baseline viewpoints and varying imaging conditions. By combining 2D texture cues with global 3D context, the proposed pipeline is well-suited for scalable street-level processing and can be used for a variety of object types. Experiments demonstrate substantially improved coverage of ground-truth sequences and more robust identity retention compared to state-of-the-art 2D-only tracking methods, achieving a 22% performance gain in challenging urban scenarios.