Search papers, labs, and topics across Lattice.
This paper introduces SAMPLe, a sharpness-aware optimizer designed to enhance prompt learning in Vision-Language Models (VLMs) by addressing the performance-generalization dilemma. By dynamically adapting to the optimization landscape, SAMPLe reduces overfitting on training data while improving generalization to unseen distributions. Experimental results across multiple prompt learning frameworks demonstrate that SAMPLe consistently outperforms existing optimizers, marking a significant advancement in the adaptability of VLMs.
SAMPLe significantly boosts the generalization of prompt learning in VLMs, outperforming traditional optimizers and mitigating overfitting issues.
Pre-trained Vision-Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for prompt learning.
