Search papers, labs, and topics across Lattice.
This paper introduces MF-VPD, a vision perception diffusion model that utilizes multimodal feature fusion to enhance discriminative visual perception tasks. By synthesizing descriptive captions through a training data generator and integrating CLIP-based textual embeddings with ViT-based visual features, MF-VPD effectively addresses the challenges of high computational costs and reliance on ground-truth image-text pairs. The model demonstrates significant improvements in depth estimation, semantic segmentation, and salient object detection, achieving an 85.17% mean intersection over union on Cityscapes while reducing model parameters by 76.7% and accelerating inference speed by 48%.
MF-VPD reduces model parameters by nearly 77% while boosting performance in visual perception tasks, making it a game-changer for efficient AI applications.
While diffusion models have demonstrated remarkable potential in generative tasks, their application to discriminative visual perception remains challenging due to high computational costs, heavy reliance on ground-truth (GT) image–text pairs, and nondeterministic outputs. To address these limitations, we propose MF-VPD, a novel vision perception diffusion model based on multimodal feature fusion (MFF). First, to overcome the scarcity of labeled image–text pairs, we introduce a training data generator that leverages BLIP to synthesize descriptive captions. Second, we design an MFF module that integrates contrastive language-image pretraining (CLIP)-based textual embeddings with Vision Transformer (ViT)-based visual features using a state-space model (SSM) strategy. This integration ensures that the diffusion backbone receives comprehensive, complementary guidance, mitigating the inadequacy of unimodal text prompts. Furthermore, we propose a lightweight visual perception adapter (VPA) to fine-tune the network while keeping the backbone frozen. Extensive experiments demonstrate MF-VPD’s superiority across depth estimation, semantic segmentation, and salient object detection. Notably, MF-VPD achieves an 85.17% mean intersection over union (mIoU) on Cityscapes and a 0.959 S-measure on DUTS while reducing error metrics by 6%–8% on NYU Depth V2 and KITTI. Crucially, compared to the VPD baseline, our adapter strategy reduces model parameters by 76.7% and accelerates inference speed by approximately 48% on depth estimation tasks. The code will be made available at https://github.com/zhx441/MF-VPD/
