Search papers, labs, and topics across Lattice.
LaVi addresses the inefficiency of visual-language integration in Large Vision-Language Models (LVLMs) by introducing a novel internal feature modulation technique. It avoids long-context expansion by injecting token-wise vision-conditioned deltas into the affine parameters of layer normalization within the LLM, directly modulating linguistic hidden states based on visual input. Experiments across 15 benchmarks demonstrate that LaVi achieves state-of-the-art multimodal performance while significantly reducing FLOPs (94.0%), improving inference speed (3.1x), and halving memory usage compared to LLaVA-OV-7B.
Forget slow visual token concatenation: LaVi modulates LLM features directly with visual context, slashing FLOPs by 94% while boosting speed and accuracy.
Despite the impressive advancements of Large Vision-Language Models (LVLMs), existing approaches suffer from a fundamental bottleneck: inefficient visual-language integration. Current methods either disrupt the model's inherent structure or introduce severe long-context computational burden, severely limiting scalability and efficiency. In this paper, we rethink multimodal integration and present LaVi, a novel LVLM that enables seamless and efficient vision-language fusion through internal feature modulation within the Large Language Models (LLMs). Unlike dominant LVLMs that rely on visual token concatenation, LaVi bypasses long-context expansion by introducing a lightweight and adaptive transformation, which incorporates visual context by injecting token-wise vision-conditioned deltas into the affine parameters of layer normalization. This mechanism directly modulates linguistic hidden states based on visual input, ensuring precise vision-language alignment while preserving the LLM's linguistic priors and drastically reducing computational costs. Extensive evaluations across 15 image and video benchmarks demonstrate that LaVi not only achieves state-of-the-art multimodal performance but also dramatically enhances efficiency. Compared to LLaVA-OV-7B, LaVi reduces FLOPs by 94.0%, improves inference speed by 3.1 times, and cuts memory usage in half - establishing LaVi as a scalable and practical solution for real-time multimodal reasoning. The code and models will be released soon.
