Search papers, labs, and topics across Lattice.
The paper introduces CLIPPER, a training-free data selection pipeline for fine-tuning multimodal LLMs that leverages in-context learning to identify informative and diverse data subsets. CLIPPER probes the MLLM with different demonstration-query combinations to assess the importance of each data point, selecting a coreset that mirrors the original dataset's perplexity distribution. Experiments on Qwen2.5-VL-7B and Llama-3.2-11B-Vision-Instruct demonstrate that CLIPPER achieves comparable performance to full fine-tuning with significantly reduced computational costs, achieving up to 47% data efficiency.
Forget expensive full fine-tuning: this training-free data selection method uses in-context learning to slash MLLM training costs while maintaining performance.
Injecting world knowledge into pretrained multimodal large language models (MLLMs) is essential for domain-specific applications. Task-specific fine-tuning achieves this by tailoring MLLMs to high-quality in-domain data but encounters scalability challenges as datasets grow, necessitating a trade-off between performance and computational overhead. Existing data selection methods rely on additional scoring models or heuristic clustering, failing to concentrate on both data importance and diversity. Moreover, both methods overlook the interplay among training samples. To address these limitations, we propose CLIPPER, a training-free data selection pipeline that separates parameter and world knowledge, and leverages in-context learning to probe model responses to different demonstration-query combinations. CLIPPER identifies coresets that mirror the original dataset's perplexity distribution, preserving critical samples while maintaining diversity. Experiments on two MLLMs and three datasets show that CLIPPER matches full fine-tuning performance with significantly lower costs: Qwen2.5-VL-7B attains 47% data efficiency on VRSBench, and Llama-3.2-11B-Vision-Instruct reduces ScienceQA training time by 37%.
