Search papers, labs, and topics across Lattice.
MAVFusion is introduced as an end-to-end video fusion framework that combines infrared and visible video by using a motion-aware sparse interaction mechanism. It leverages optical flow to identify dynamic regions and applies computationally intensive cross-modal attention to these areas, while using a lightweight interaction module for static regions. Experiments on multiple benchmarks show MAVFusion achieves state-of-the-art performance with a speed of 14.16 FPS at 640x480 resolution, demonstrating improved efficiency and fusion quality.
Fusing infrared and visible video doesn't have to be slow: MAVFusion achieves state-of-the-art results at 14 FPS by focusing cross-modal attention only on motion-rich regions.
Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16\,FPS at $640 \times 480$ resolution. The source code will be available at https://github.com/ixilai/MAVFusion.
Tsinghua AI