Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates whether LLMs and LVLMs share neurons during multi-step inference, finding a significant overlap in top-activated neurons, suggesting a modality-invariant inference subspace. They causally probe these shared neurons, demonstrating their functional role in encoding consistent concept-level effects. Based on these findings, they propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient transfer learning framework that leverages shared neurons to enhance LVLM inference by injecting low-rank updates within the shared-neuron subspace, achieving improved performance on mathematics and perception benchmarks without large-scale fine-tuning.
LLMs and LVLMs share more than half their top-activated neurons during multi-step inference, opening a surprisingly cheap path to boost vision-language reasoning by transplanting skills from text-only models.
Large vision-language models (LVLMs) have rapidly advanced across various domains, yet they still lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making. Motivated by their shared transformer architectures, we investigate whether the two model families rely on common internal computation for such inference. At the neuron level, we uncover a surprisingly large overlap: more than half of the top-activated units during multi-step inference are shared between representative LLMs and LVLMs, revealing a modality-invariant inference subspace. Through causal probing via activation amplification, we further show that these shared neurons encode consistent and interpretable concept-level effects, demonstrating their functional contribution to inference. Building on this insight, we propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs. SNRF profiles cross-model activations to identify shared neurons, computes a low-rank approximation of inter-model weight differences, and injects these updates selectively within the shared-neuron subspace. This mechanism strengthens multimodal inference performance with minimal parameter changes and requires no large-scale multimodal fine-tuning. Across diverse mathematics and perception benchmarks, SNRF consistently enhances LVLM inference performance while preserving perceptual capabilities. Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models. Our code is available at [https://github.com/chenhangcuisg-code/Do-LLMs-VLMs-Share-Neurons](https://github.com/chenhangcuisg-code/Do-LLMs-VLMs-Share-Neurons).