Search papers, labs, and topics across Lattice.
This paper introduces the Parallel Vision-Language (ParVL) framework, which optimizes the allocation of computation between Vision Transformer and Large Language Model components in Multimodal Large Language Models (MLLMs) by reusing existing backbone parameters. By implementing branch-specific prefix parameters and conducting end-to-end training on approximately 13 billion tokens, the authors systematically explore the trade-offs in computation allocation, demonstrating that task-specific optimization can significantly enhance multimodal performance. The results show that the optimal allocation between vision and language varies across tasks, challenging the conventional fixed computation approach in MLLMs.
Task-specific optimization in multimodal models can significantly boost performance by dynamically reallocating shared computation resources between vision and language components.
Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.
