Antonio Carlos Rivera

EDP University of Puerto Rico: San Sebastian Abstract Large Language Models (LLMs) and Vision-Language Large Models (LVLMs) have achieved remarkable progress in natural language processing and multimodal understanding. Despite their impressive generalization capabilities, current LVLMs often exhibit insufficient robustness, proneness to hallucination, and reasoning errors in complex real-world scenarios, particularly when precise image region localization and fine-grained visual reasoning are required. To address these limitations, we propose the Hierarchical Contextual Grounding LVLM (HCG-LVLM), a novel architecture that mimics human coarse-to-fine cognitive processing. HCG-LVLM employs a two-layered approach: a Global Contextual Perception layer for initial broad understanding and a Fine-grained Local Grounding layer. The latter incorporates a Local Detail Enhancement Module to extract high-resolution features and a Semantic Consistency Validator to ensure accurate, hallucination-free visual-language alignment. Through an adaptive fusion mechanism, information from both layers is integrated for robust and precise outputs. Extensive experiments on challenging datasets, including GQA, A-OKVQA for fine-grained VQA, and RefCOCO/+/g for Referring Expression Comprehension, demonstrate that HCG-LVLM consistently outperforms state-of-the-art models such as Flamingo, BLIP-2, and MiniGPT-4. Our model achieves superior accuracy and significantly reduces hallucination, validating the effectiveness of its hierarchical design in enhancing fine-grained visual-language understanding and precise grounding capabilities. I Introduction In recent years, large language models (LLMs) and vision-language large models (LVLMs) have demonstrated remarkable progress in the fields of natural language processing and multimodal understanding, respectively [1]. These powerful models, often leveraging visual in-context learning [2], have exhibited impressive generalization capabilities [3] and exceptional performance across a wide array of tasks, including question answering, summarization, image captioning, story ending generation [4], and visual question answering (VQA) [5]. Their ability to process and synthesize information from vast datasets has opened up new avenues for intelligent systems, making them central to the advancement of artificial intelligence. Despite their significant achievements and powerful pre-training on massive datasets, current LVLMs still face considerable challenges when deployed in real-world scenarios. Specifically, they often suffer from insufficient robustness, prone to generating hallucinations, or making erroneous inferences when confronted with complex, long-tail distributed data, or tasks requiring fine-grained visual cues [6]. This limitation is particularly pronounced in specialized domains like medicine, where models require abnormal-aware feedback for improvement [7], and in multimodal tasks that necessitate precise image region localization and detailed visual reasoning, where existing LVLMs may struggle to capture subtle visual nuances or comprehend intricate spatial relationships. Addressing chaotic contexts is also a critical challenge for enhancing their reasoning and robustness [8]. This research is motivated by the critical need to address these limitations, aiming to enhance the robustness and accuracy of LVLMs, especially in tasks demanding sophisticated fine-grained visual information processing and complex spatial reasoning. We propose to tackle these issues through a novel hierarchical attention and knowledge-enhanced mechanism designed to enable LVLMs to more effectively integrate global contextual understanding with local details, thereby mitigating the occurrence of hallucinations. To this end, we introduce a novel model architecture named Hierarchical Contextual Grounding LVLM (HCG-LVLM), which is specifically engineered to improve performance in fine-grained visual understanding and precise visual-language grounding tasks. Inspired by the human cognitive process of moving from coarse to fine details, HCG-LVLM processes visual-language information across two primary hierarchical layers. The first layer, termed Global Contextual Perception, utilizes an existing pre-trained LVLM (e.g., based on a ViT-Encoder and LLM-Decoder architecture) to process input images and text queries. This stage focuses on understanding the overall image content, identifying primary objects, and capturing the macroscopic semantics of the query, thereby establishing an initial visual-language correspondence and generating a coarse-grained attention map or region proposals. Building upon this, the second layer, Fine-grained Local Grounding, dynamically focuses on key regions within the image based on the coarse-grained attention from the first layer. This layer incorporates a Local Detail Enhancement Module to extract higher-resolution, more discriminative local visual features from these focused regions. Concurrently, a Semantic Consistency Validator is designed to integrate local visual features with the original text query, employing contrastive learning or consistency loss functions to ensure accurate and hallucination-free visual-language correspondence at the local level. For instance, if a query involves ”a small red cup on the table,” the first layer might identify ”table” and ”cup,” while the second layer would zoom into the ”cup” area to verify its color and size against ”small red.” Ultimately, information from both layers is adaptively fused to produce the final answer or localization result, significantly enhancing the model’s ability to discern subtle differences and precisely follow instructions in complex scenarios. Our proposed HCG-LVLM is built upon a robust foundation, employing a pre-trained LVLM based on the ViT-L/14 and LLaMA-