Search papers, labs, and topics across Lattice.
This paper introduces Cross-Modal Visual Feedback (CMVF) to enhance automatic prompt optimization (APO) for vision-language models (VLMs) by incorporating visual input during the optimization process. By allowing the optimizer to analyze failed predictions in the context of the corresponding input images, CMVF addresses the limitations of traditional APO methods that rely solely on textual feedback. The results show that CMVF outperforms existing baselines across 12 visual question answering datasets, achieving an average improvement of 2.4 points and up to 6.5 points on specific benchmarks, while maintaining the efficiency of text-only prompts during inference.
Incorporating visual feedback into prompt optimization leads to significant performance improvements in vision-language tasks, revealing previously unrecognized error patterns.
Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks, the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks. Moreover, the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization.
