Search papers, labs, and topics across Lattice.
The paper introduces Entropy-Gradient Grounding (EGG), a training-free method for improving evidence retrieval in VLMs by using the entropy of the next-token distribution as a signal to identify relevant visual regions. EGG backpropagates this entropy to visual token embeddings, creating relevance maps that guide the model to focus on ambiguous or detail-critical areas in the image. Experiments across seven benchmarks and four VLMs show that EGG consistently improves performance, especially in detail-critical and high-resolution scenarios, while also enhancing interpretability.
Forget training wheels: this training-free method leverages uncertainty to guide vision-language models to the right image regions, boosting performance on detail-oriented tasks.
Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.
