Search papers, labs, and topics across Lattice.
The paper introduces FewMMBench, a new benchmark to evaluate the few-shot learning capabilities of multimodal large language models (MLLMs) using In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. The benchmark covers a diverse set of multimodal tasks, including attribute recognition and temporal reasoning, enabling systematic analysis across various model families and prompting strategies. Experiments on 26 open-weight MLLMs reveal that instruction-tuned models show strong zero-shot performance but exhibit limited or negative gains from few-shot examples or CoT prompting, suggesting areas for improvement in few-shot multimodal learning.
Instruction-tuned MLLMs, despite excelling in zero-shot scenarios, surprisingly fail to leverage few-shot examples or chain-of-thought prompting, suggesting a fundamental limitation in their ability to learn from demonstrations.
As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench
