Search papers, labs, and topics across Lattice.
VLMaterial is a training-free framework that fuses vision-language models (VLMs) with radar data for improved material identification, addressing limitations of vision-only and existing camera-radar fusion methods. It uses a dual-pipeline architecture, combining a VLM-based optical pipeline with a radar pipeline that extracts the dielectric constant of materials. By incorporating a context-augmented generation strategy to inject radar-specific physical knowledge into the VLM and adaptively fusing the sensor outputs, VLMaterial achieves 96.08% accuracy in real-world experiments, matching closed-set benchmarks without task-specific training.
Forget training data: VLMaterial achieves state-of-the-art material recognition by cleverly fusing VLMs with radar data and physics-based reasoning.
Accurate material recognition is a fundamental capability for intelligent perception systems to interact safely and effectively with the physical world. For instance, distinguishing visually similar objects like glass and plastic cups is critical for safety but challenging for vision-based methods due to specular reflections, transparency, and visual deception. While millimeter-wave (mmWave) radar offers robust material sensing regardless of lighting, existing camera-radar fusion methods are limited to closed-set categories and lack semantic interpretability. In this paper, we introduce VLMaterial, a training-free framework that fuses vision-language models (VLMs) with domain-specific radar knowledge for physics-grounded material identification. First, we propose a dual-pipeline architecture: an optical pipeline uses the segment anything model and VLM for material candidate proposals, while an electromagnetic characterization pipeline extracts the intrinsic dielectric constant from radar signals via an effective peak reflection cell area (PRCA) method and weighted vector synthesis. Second, we employ a context-augmented generation (CAG) strategy to equip the VLM with radar-specific physical knowledge, enabling it to interpret electromagnetic parameters as stable references. Third, an adaptive fusion mechanism is introduced to intelligently integrate outputs from both sensors by resolving cross-modal conflicts based on uncertainty estimation. We evaluated VLMaterial in over 120 real-world experiments involving 41 diverse everyday objects and 4 typical visually deceptive counterfeits across varying environments. Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks while eliminating the need for extensive task-specific data collection and training.
