Search papers, labs, and topics across Lattice.
This paper introduces the Dual Modality Image-Prompted Diffusion Model (DIDM) for zero-shot hyperspectral pansharpening, which effectively integrates low-resolution hyperspectral and panchromatic images through cross-attention mechanisms. By encoding these images into spectral and spatial prompt tokens, DIDM allows for direct interaction with a frozen remote sensing diffusion model, enhancing both spatial detail and spectral fidelity. Experimental results show that DIDM outperforms existing methods across multiple datasets, achieving the highest quality normalized root mean square error (HQNR) in full resolution evaluations.
Internal dual modality prompting can significantly enhance spatial detail while preserving spectral fidelity in hyperspectral pansharpening.
Hyperspectral pansharpening aims to reconstruct a high resolution hyperspectral (HRHS) image from a panchromatic (PAN) image and a low resolution hyperspectral (LRHS) image while preserving both spatial details and spectral fidelity. Recent diffusion based methods exploit pretrained image priors by generating a low dimensional representation and subsequently mapping it to the hyperspectral domain. However, the observed panchromatic and hyperspectral images are typically imposed only through external reconstruction objectives, limiting their direct interaction with the diffusion prior. To address this issue, we propose dual-modality image-prompted diffusion model (DIDM) for zero shot hyperspectral pansharpening. DIDM encodes the low resolution hyperspectral and panchromatic observations into spectral and spatial prompt tokens, respectively, and injects them into intermediate features of a frozen remote sensing diffusion model through cross attention, allowing complementary spectral and spatial information to directly guide diffusion feature evolution. In addition, we introduce a panchromatic guided weighted pixel aware total variation regularizer that combines low resolution hyperspectral degradation fidelity and panchromatic response fidelity with gradient adaptive structural regularization, thereby preserving structural discontinuities while suppressing spurious variations in homogeneous regions. Extensive experiments on Pavia, Chikusei, and Houston under reduced resolution protocols show that DIDM achieves the best performance across all evaluated metrics, while full resolution evaluation on FR1 yields the highest HQNR among the compared methods. These results demonstrate that internal dual modality prompting and panchromatic guided structural regularization provide an effective balance between spatial detail enhancement and spectral preservation.
