Search papers, labs, and topics across Lattice.
This paper challenges the assumption that language pre-trained models are unsuitable for vision tasks due to differing parameter spaces. It introduces "random label bridge training," a modality adaptation technique that aligns LLM parameters with vision tasks without manual labeling. The key finding is that partial bridge training, leveraging inherent foundational properties in certain LLM layers, often outperforms full fine-tuning for visual tasks.
LLMs can be surprisingly effective vision models, even without full fine-tuning, thanks to inherent foundational properties in specific layers that transfer via a simple "random label bridge training" technique.
The ratio of outlier parameters in language pre-training models and vision pre-training models differs significantly, making cross-modality (language and vision) inherently more challenging than cross-domain adaptation. As a result, many prior studies have focused on cross-domain transfer rather than attempting to bridge language and vision modalities, assuming that language pre-trained models are unsuitable for downstream visual tasks due to disparate parameter spaces. Contrary to this assumption, we show that adding a bridge training stage as a modality adaptation learner can effectively align Large Language Model (LLM) parameters with vision tasks. Specifically, we propose a simple yet powerful solution random label bridge training that requires no manual labeling and helps LLM parameters adapt to vision foundation tasks. Moreover, our findings reveal that partial bridge training is often advantageous, as certain layers in LLMs exhibit strong foundational properties that remain beneficial even without fine-tuning for visual tasks. This surprising discovery opens up new avenues for leveraging language pre-trained parameters directly within vision models and highlights the potential of partial bridge training as a practical pathway to cross-modality adaptation.
