Search papers, labs, and topics across Lattice.
To overcome superficial imitation in multimodal in-context learning (ICL), the authors develop a self-refinement framework that reformulates demonstrations as explicit contrastive pairs of suboptimal and corrected responses along with their underlying reasoning deltas. This directly addresses the failure of MLLMs to deduce complex visual reasoning paths when merely conditioned on standard input-output demonstration pairs. Coupled with a response-conditioned retrieval mechanism and an alignment stopping controller, the approach achieves consistent downstream gains across multimodal tasks, with particularly significant improvements on complex visual question answering.
Instead of mimicking gold-standard examples, MLLMs reason significantly better when shown suboptimal outputs alongside explicit contrastive paths to correct them.
In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).
