Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MAViE, a Multi-scale Adaptive Vision Encoder that enhances visual perception and multimodal reasoning by effectively fusing features from different layers of a vision Transformer. By employing position-dependent gates and question-conditioned token routing, MAViE preserves essential visual information while significantly reducing the number of tokens processed, leading to improved performance on various visual question answering benchmarks. The method achieves an 80% reduction in visual tokens and a 2.2 percentage point increase in accuracy across multiple datasets, while also decreasing inference latency.
MAViE slashes visual token counts by 80% while boosting accuracy on key benchmarks, revolutionizing how we approach multimodal reasoning.
Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0\%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228\,ms to 129\,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.