Search papers, labs, and topics across Lattice.
This paper introduces FusionRS, the first large-scale RGB-infrared-text dataset tailored for dual-modal vision-language learning in remote sensing, addressing the underutilization of infrared data in existing models. By constructing aligned RGB-IR image pairs with both conventional and IR-aware captions, the authors enable enhanced understanding of infrared-specific visual properties. Experiments demonstrate that models trained on FusionRS significantly outperform those trained solely on RGB data, particularly in RGB-IR alignment and dual-modal captioning tasks.
Infrared data, often overlooked, can dramatically enhance vision-language models, as shown by FusionRS's ability to improve dual-modal understanding and captioning performance.
Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored. Infrared images provide distinctive cues, including thermal intensity structures, object boundaries, and illumination-invariant scene features, which can enrich visual-language learning beyond conventional RGB observations. However, a large-scale RGB-infrared-text dataset for remote sensing vision-language modeling is still absent. To address this gap, we introduce FusionRS, the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing. FusionRS is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs. Each pair is associated with conventional scene captions and IR-aware captions that explicitly describe infrared-specific visual properties while preserving semantic content. Based on FusionRS, we train dual-modal vision-language foundation models for RGB-IR joint understanding. We first train CLIP-style models for RGB-IR-text alignment, and then fine-tune generative VLMs for dual-modal RGB-IR captioning. Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings. Ablation studies further verify that IR-aware captions are crucial for strengthening infrared-language alignment, highlighting the importance of modality-specific textual supervision for more scalable RGB-infrared remote sensing vision-language representation learning.
