Search papers, labs, and topics across Lattice.
This paper investigates why multimodal in-context learning (ICL) underperforms compared to text-only ICL, finding that while zero-shot performance is comparable, few-shot multimodal ICL degrades significantly. Through a decomposition of ICL into task mapping construction and transfer, the authors reveal a lack of reasoning-level alignment between visual and textual representations, hindering reliable task mapping transfer. They propose and validate a simple inference-stage enhancement method to reinforce task mapping transfer, improving multimodal ICL performance.
Multimodal LLMs struggle with in-context learning not because they can't see, but because they can't reason across modalities, leading to a breakdown in transferring learned task mappings.
In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available \href{https://github.com/deeplearning-wisc/Multimocal-ICL-Analysis-Framework-MGI}{here}.
