Search papers, labs, and topics across Lattice.
This paper introduces BeyondFusion, a novel latent diffusion framework designed for calibration-free infrared super-resolution and infrared-visible fusion, addressing the challenges posed by cross-sensor misalignment. By integrating a cross-modal self-aligning (CMSA) module within a denoising U-Net, the framework learns content-adaptive correspondences without the need for explicit geometric registration. Extensive experiments demonstrate that BeyondFusion achieves superior performance in generating high-quality fused images and infrared reconstructions under various conditions, including misalignments and real-world captures.
BeyondFusion achieves high-quality infrared-visible fusion without the need for calibration, leveraging self-alignment to overcome sensor misalignment challenges.
Mobile infrared-visible imaging typically pairs a compact infrared sensor with a high-resolution visible camera for complementary perception. While cross-sensor misalignment caused by different optics, viewpoints, fields of view, and exposure timings hinders practical deployment. In this paper, we propose BeyondFusion, a unified latent diffusion framework for calibration-free visible-guided infrared super-resolution and infrared-visible fusion tasks. The proposed framework supports both task-specific training and joint training where two tasks are optimized and executed as two readouts of the same generative process. Instead of relying on explicit registration or geometric warping, BeyondFusion introduces a cross-modal self-aligning (CMSA) module into the denoising U-Net. CMSA reorganizes infrared and visible latent tokens into a shared attention space to learn content-adaptive cross-modal correspondence during the denoising process. Together with misalignment augmentation module, the model is facilitated to exploit visible structural and semantic cues while preserving thermal consistency, enabling high-frequency infrared reconstruction and informative fused-image generation under uncalibrated conditions. Extensive experiments on public benchmarks and a mobile infrared-visible imaging system show strong performance across aligned inputs, low-resolution infrared observations, synthetic misalignments, and real mobile captures with unsynchronized sensors. Ablation studies, unified training analysis, and downstream pedestrian detection further validate the effectiveness of BeyondFusion for calibration-free multimodal imaging.
Tsinghua AI