Search papers, labs, and topics across Lattice.
The paper introduces Fusion to Enhance (FtZ), a novel vision tower framework for MLLMs that combines a semantically powerful anchor encoder with a perception-rich augmenting encoder using multi-head cross-attention. This addresses the limitation of single-encoder MLLMs that struggle with tasks requiring fine-grained visual detail. FtZ achieves significant performance gains on benchmarks like TextVQA, POPE, MMMU, MME and MM-Vet, demonstrating the effectiveness of composing heterogeneous expert encoders to improve visual perception in MLLMs.
MLLMs can get a serious vision boost by fusing features from multiple specialized visual encoders, rather than relying on a single, semantically-focused one.
Multimodal Large Language Models (MLLMs) have made significant progress in bridging visual perception with high-level textual reasoning. However, they face a fundamental contradiction: while excelling at complex semantic understanding, these models often fail at basic visual tasks that require precise detail perception. This deficiency primarily stems from the prevalent architectural reliance on a single vision encoder optimized for high-level semantic alignment, which inherently sacrifices the ability to capture fine-grained visual information. To address this issue, we introduce Fusion to Enhance (FtZ), a novel vision tower framework. FtZ moves beyond the single-encoder design by innovatively composing a semantically powerful anchor encoder with a perception-rich augmenting encoder via a lightweight Multi-Head Cross-Attention mechanism. Experimental results demonstrate that on several challenging benchmarks demanding fine-grained visual understanding, such as TextVQA, POPE, MMMU, MME and MM-Vet, our FtZ model significantly outperforms baselines that use only a single encoder or existing feature fusion methods. This work proves that composing heterogeneous expert encoders is an efficient and effective path to overcoming the visual perception bottleneck in current MLLMs, offering a new design paradigm for building next-generation AI systems with stronger perceptual capabilities.
