Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the degradation of text capabilities in language models when fine-tuned into vision-language models, identifying a phenomenon termed "attention-sink corruption" that affects performance on tasks requiring precise output adherence. The authors introduce a metric called Sink Strength, which predicts the extent of capability loss in the language model post-adaptation based solely on its pre-fine-tuning characteristics. Their findings reveal that traditional methods for mitigating this degradation are ineffective, emphasizing the importance of using Sink Strength as a diagnostic tool prior to vision-language training.
Fine-tuning a language model into a vision-language model can lead to significant text capability loss, with a novel metric predicting degradation before adaptation.
Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.