Search papers, labs, and topics across Lattice.
This paper introduces Decoupled Visual Processing (DVP), a novel framework that enhances multimodal large language models (MLLMs) by substituting the upper decoder layers with a lightweight transformer block specifically for visual token processing. By allowing visual and textual tokens to be processed independently after shared initial layers, DVP significantly reduces the number of trainable parameters while maintaining competitive performance on benchmarks such as MME, POPE, and ChartQA. The findings indicate that a decoupled approach can effectively optimize visual representation learning in MLLMs, making fine-tuning more efficient and less resource-intensive.
Achieving competitive multimodal performance while training only a fraction of the parameters reveals a new pathway for efficient visual processing in large language models.
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture. However, fine-tuning all parameters of these models for visual instruction tuning is computationally expensive and often unnecessary, as the representation requirements for visual and textual tokens diverge significantly in the deeper layers of the network. In this paper, we propose Decoupled Visual Processing (DVP), an efficient training framework that replaces the upper decoder layers of a pretrained LLM with a lightweight, independently trainable single transformer block dedicated exclusively to visual token processing. Specifically, after shared processing through the first half of the decoder layers, visual and textual tokens are split: visual tokens are routed through a newly initialized single transformer block while textual to-kens continue through the original frozen de-coder layers. The two streams are then concatenated before the language modeling head. During training, only the single transformer block is updated, dramatically reducing the number of trainable parameters. Experiments on the LLaVA-1.5 framework demonstrate that DVP achieves competitive performance on MME, POPE, and ChartQA benchmarks while training only a fraction of the total parameters, suggesting that visual representations in MLLMs can be effectively learned through a decoupled, parameter-efficient pathway.
Tsinghua AI