Search papers, labs, and topics across Lattice.
This study introduces NeuPAT, a novel framework that addresses the challenge of preserving language capabilities in large language models (LLMs) during multimodal expansion. By analyzing the heterogeneous plasticity of neurons in pretrained LLMs, the authors identify which neurons are critical for language preservation and which can adapt to multimodal tasks. NeuPAT effectively recovers 94.5% of the language capability loss from standard tuning while maintaining strong multimodal performance across various LLM families.
NeuPAT recovers nearly all lost language capabilities in multimodal LLMs while ensuring robust performance across diverse tasks.
Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.
