Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Entity Exploration (HEE), a novel framework designed to enhance high-resolution image perception in multimodal large language models (MLLMs) without requiring prior training. By employing a dual scoring mechanism to evaluate image regions and a confidence-guided backtracking approach, HEE dynamically explores fine-grained entities and organizes them into a multi-level semantic hierarchy. The method significantly outperforms existing training-free approaches like ZoomEye and RAP in accuracy and efficiency across multiple complex benchmarks, showcasing its robustness and generalization capabilities.
HEE transforms static image understanding into a dynamic, query-guided exploration that significantly boosts accuracy in high-resolution perception tasks.
High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs), as fine-grained details are often lost when the image is processed as a whole. Existing methods either require training to teach models where to look or heuristically divide the image into fixed regions, both of which struggle to generalize in complex HR scenes. In this work, we propose Hierarchical Entity Exploration (HEE), a training-free and model-agnostic framework that transforms static image understanding into dynamic, query-guided entity exploration. HEE first evaluates each region using a dual scoring mechanism to determine whether it already contains sufficient evidence to answer the question. If not, it applies object detection within the most promising region to extract fine-grained entities, clusters them into coherent subregions, and organizes them into a multi-level semantic hierarchy for deeper exploration. When deeper regions still fail to yield confident answers, a confidence-guided backtracking mechanism revisits alternative paths to ensure adaptive perception. Extensive results show that HEE outperforms training-free methods like ZoomEye and RAP in both accuracy and efficiency on two complex HR benchmarks (Visual Probe and HR-Bench), across different MLLMs such as Qwen2.5-VL and LLaVA-OneVision. Moreover, HEE demonstrates generalization on the MME-RealWorld benchmark.
DAMO