Search papers, labs, and topics across Lattice.
This paper introduces ENCORE, an entropy-guided framework designed to enhance the performance of Vision-Language Models (VLMs) by addressing the limitations of fixed-resolution sub-image cropping. By leveraging the correlation between early-layer image-text entropy and task accuracy, ENCORE employs an Entropy-based Cropping Strategy (ECS) during inference to select optimal image crops and utilizes Entropy Regularization Training (ERT) to improve attention on relevant visual tokens during training. The method demonstrates a notable average accuracy gain of 1.43% across ten VQA benchmarks while fine-tuning only 0.14% of parameters, achieving state-of-the-art results among 2B-parameter VLMs.
Fine-tuning just 0.14% of parameters, ENCORE boosts VLM accuracy by 1.43% through innovative entropy-guided cropping and attention techniques.
Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text entropy of cross-modal attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose \textbf{ENCORE}, an entropy-guided framework with two components: At inference, an \textbf{Entropy-based Cropping Strategy} (ECS) evaluates a small set of candidate crops and selects the one with minimal entropy, preserving contiguous regions relevant to the prompt. At training, \textbf{Entropy Regularization Training} (ERT) augments next-token prediction with an entropy term that sharpens attention on key visual tokens while down-weighting irrelevant ones. Experiments on ten VQA benchmarks show that ENCORE, fine-tuning only 0.14\% of parameters, achieves an average 1.43\% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.
