Search papers, labs, and topics across Lattice.
This paper introduces LookBack, a novel method for scoring responses from Large Vision-Language Models (LVLMs) by incorporating a visual lookback score that assesses how well response tokens reference image tokens. The authors identify that existing confidence-based metrics fail to account for visual grounding, as they predominantly measure textual plausibility rather than alignment with the visual input. Through extensive benchmarking across multiple models, LookBack demonstrates a consistent improvement in response selection accuracy with minimal computational overhead.
Existing scoring methods for LVLMs miss the mark, but LookBack reveals how visual grounding can drastically enhance response quality.
Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
