Search papers, labs, and topics across Lattice.
This paper addresses the issue of modality imbalance in unified multimodal models (UMMs) during instruction tuning, where language gradients often overshadow image gradients, leading to subpar image generation quality. By reformulating the tuning process as a bi-objective optimization problem, the authors introduce Pareto LoRA, a method that optimally integrates gradients from both modalities to enhance balance in multimodal performance. Experiments reveal that Pareto LoRA significantly improves perceptual image quality by up to 44.9% compared to traditional LoRA, while preserving text generation quality.
Achieving up to 44.9% improvement in image quality without sacrificing text performance reveals a new frontier in multimodal model optimization.
Unified multimodal models (UMMs) have recently emerged as a promising paradigm for integrating multimodal understanding and generation within a single autoregressive transformer. However, during multimodal instruction tuning, these models often exhibit pronounced modality imbalance: language gradients dominate optimization, thus leading to lower image generation quality, especially under parameter-efficient fine-tuning such as LoRA. In this work, we systematically analyze modality imbalance in LoRA-based fine-tuning of UMMs for interleaved text-image generation. We show that vision modality performance degrades substantially more than text modality performance when compared to unimodal counterparts, and that modality-specific gradients can differ by orders of magnitude across various tasks and layers. Motivated by this observation, we reformulate the multimodal instruction tuning as a bi-objective optimization problem and propose Pareto LoRA, a Pareto-optimal gradient integration strategy that balances the text and image objectives by modulating the gradient direction and strength. Experiments on the CoMM benchmark with Emu2 demonstrate that Pareto LoRA consistently improves multimodal generation balance, achieving up to 44.9% gains in perceptual image quality over vanilla LoRA while maintaining comparable text performance.
