Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces MAGIC, a training-free coreset selection method for vision-language models (VLMs) that identifies compact, high-quality training subsets. MAGIC leverages three signals from a pretrained VLM: Multimodal Gain, Bridging Relevance, and Skill-Neuron Signatures, to filter low-quality examples, rank candidates, and allocate a budget across neuron signatures. Experiments on LLaVA-665K and Vision-Flan datasets show that MAGIC achieves comparable performance to full finetuning with only 20% of the data, while significantly reducing training time.
Get 5x faster VLM training without sacrificing performance: MAGIC distills massive multimodal datasets down to the most behaviorally-rich 20% using intrinsic signals from the pretrained model itself.
Instruction tuning of large vision-language models (LVLMs) increasingly depends on massive multimodal corpora, yet these datasets contain samples with substantial redundancy, low visual dependency, and highly imbalanced coverage of multimodal reasoning behaviors. As a result, uniform subsampling or naive score-based selection often yields suboptimal training subsets. We introduce MAGIC, a training-free, forward-only coreset selection method designed to construct compact yet behaviorally faithful subsets for multimodal instruction tuning. MAGIC is built on three intrinsic signals extracted from a pretrained VLM: Multimodal Gain, which measures the likelihood improvement obtained from visual input; Bridging Relevance, which captures the sharpness of answer-token grounding over visual tokens; and Skill-Neuron Signatures, which characterize the functional computation elicited by each sample via top-activated feed-forward neurons. MAGIC combines these signals in a three-stage pipeline: filtering low-gain examples, ranking candidates by a normalized quality objective, and performing bucket-wise budget allocation over discrete neuron signatures to preserve latent multimodal skill coverage. This formulation avoids backpropagation, auxiliary selector training, and expensive clustering in continuous activation spaces, while remaining efficient and easily deployable in existing VLMs. Across LLaVA-665K and Vision-Flan datasets, and transfer settings to large target models, LLaVA-1.5-7B and -13B, MAGIC consistently improves over strong baselines under matched 20% budgets: it achieves 100.3% relative performance to full finetuning on LLaVA-665K and 101.6% relative performance on Vision-Flan-186K, while yielding a 73.7% reduction in wall-clock run time.