Search papers, labs, and topics across Lattice.
This paper introduces SPOT-E, a novel test-time method that enhances the performance of frozen vision-language models (VLMs) by utilizing question-conditioned visual spotlights to improve evidence readout in evidence-intensive tasks. By employing an entropy-shaping objective that incorporates low-entropy anchors, SPOT-E effectively reduces answer uncertainty while maintaining high-confidence token integrity. The method demonstrates consistent performance gains and robustness against visual corruptions across various benchmarks and VLM families, highlighting its practical utility in real-world applications.
SPOT-E transforms frozen VLMs into more reliable evidence readers by dynamically spotlighting critical visual information during inference.
Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: \url{https://github.com/YinBo0927/SPOT-E}
NUS