Search papers, labs, and topics across Lattice.
This paper introduces Talk2Sensors, a novel multi-sensor 3D visual grounding dataset that integrates camera, LiDAR, and 4D radar inputs to enhance outdoor perception in autonomous driving. The authors develop TSFormer, a Transformer-based framework that employs a coarse-to-fine fusion strategy to dynamically route sensor-specific cues based on linguistic prompts, addressing the limitations of existing methods that rely solely on monocular images. Experimental results show that TSFormer outperforms the best baseline by 8.05 mAP on the Talk2Sensors dataset and achieves a 53.05% accuracy on the Mono3DRefer benchmark, demonstrating its effectiveness in real-world applications.
TSFormer redefines 3D visual grounding by leveraging diverse sensor modalities, achieving significant performance gains in outdoor autonomous driving scenarios.
As a key capability for embodied intelligence, 3D visual grounding (3DVG) has been predominantly studied in indoor scenes with RGB-D or point-cloud inputs, while existing outdoor extensions largely rely on monocular images alone. Both settings fall short of real-world outdoor perception, where heterogeneous sensors capture complementary yet distinct physical properties, such as visual texture, 3D geometry, and object kinematics, that are indispensable for flexible and robust query-adaptive grounding but remain under-exploited. To bridge this gap, we introduce Talk2Sensors, the first multi-sensor 3D visual grounding dataset built upon camera, LiDAR, and 4D radar. It contains 8,682 language instructions and 20,558 referred objects, with diverse prompts explicitly aligned with sensor-specific physical cues. Furthermore, we propose TSFormer, a unified Transformer-based framework for language-guided 3D visual grounding in autonomous driving. TSFormer adopts a coarse-to-fine property-aware fusion strategy: the Language-Routed Property Sampler first performs coarse text-conditioned feature retrieval by modulating sensor sampling weights with query-level linguistic cues, while the subsequent Sparse-Preserving Modality Arbiter module conducts fine-grained modality arbitration and text-guided refinement to determine the precise referred spatial location. This design enables dynamic routing of appearance, geometry, and motion cues according to the semantic requirements of each prompt, preventing dense modalities from overwhelming sparse but critical sensor signals. Extensive experiments demonstrate that TSFormer achieves state-of-the-art performance across multiple benchmarks: it improves over the strongest baseline by 8.05 mAP on Talk2Sensors, and transfers to the monocular Mono3DRefer benchmark with 53.05\% Acc@0.5.
