Search papers, labs, and topics across Lattice.
This paper addresses the limitations of uniform patch-wise contrastive learning in visible-infrared (VIS-IR) pre-training by introducing Importance-Aware Sampling (IAS), which prioritizes training on more reliable patches based on infrared structural cues. By adjusting the contrastive objective to emphasize these reliable patches and employing a curriculum learning strategy, IAS significantly enhances representation learning and downstream task performance. Extensive experiments across various VIS-IR benchmarks reveal that IAS consistently outperforms strong baselines in tasks such as semantic segmentation and cross-modal retrieval.
Importance-Aware Sampling (IAS) reveals that not all patches in VIS-IR data are created equal, leading to substantial performance gains in multi-sensor perception tasks.
Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.
