Search papers, labs, and topics across Lattice.
This paper addresses the granularity dilemma in vision-language models by introducing UPrompt, a U-shaped multi-granularity prompt learning framework that integrates both coarse and fine representations. By leveraging a structure inspired by U-Net, UPrompt enhances global context propagation and local detail refinement through parallel representations in visual and textual modalities. Experimental results demonstrate that UPrompt significantly outperforms existing methods on multiple benchmarks, achieving notable improvements in generalization while maintaining efficiency.
UPrompt achieves a remarkable 7.3 rSum improvement over existing models on MSCOCO, redefining the standards for cross-task generalization in vision-language tasks.
The prompt learning paradigm for vision-language models is effective yet faces a granularity dilemma: global prompts lack fine-grained semantic awareness, while local prompts ignore contextual associations, limiting cross-task generalization. This dilemma exists in dense prediction tasks. Inspired by U-Net, which unifies multi-level representations across granularities, we propose UPrompt, a U-shaped multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global context to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, surpasses CoCoA-Mix by 5.09% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).
