Search papers, labs, and topics across Lattice.
This paper investigates the necessity of fine-tuning the backbone of multimodal large language models (MLLMs) by focusing solely on training the projector that connects the backbone to modality-specific encoders. The authors demonstrate that projector-only training achieves competitive multimodal performance while preserving the language model's existing capabilities and significantly increasing training throughput. Their findings challenge conventional approaches, revealing that joint training can lead to performance drift and inefficiencies in adapting to new modalities.
Training only the projector can match or exceed the performance of fully fine-tuned multimodal models while avoiding capability drift.
The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.
