Mateo Alejandro Rojas

Technological University of Peru Abstract Complex Visual Question Answering (Complex VQA) tasks, which demand sophisticated multi-modal reasoning and external knowledge integration, present significant challenges for existing large vision-language models (LVLMs) often limited by their reliance on high-level global features. To address this, we propose MV-CoRe (Multimodal Visual-Conceptual Reasoning), a novel model designed to enhance Complex VQA performance through the deep fusion of diverse visual and linguistic information. MV-CoRe meticulously integrates global embeddings from pre-trained Vision Large Models (VLMs) and Language Large Models (LLMs) with fine-grained semantic-aware visual features, including object detection characteristics and scene graph representations. An innovative Multimodal Fusion Transformer then processes and deeply integrates these diverse feature sets, enabling rich cross-modal attention and facilitating complex reasoning. We evaluate MV-CoRe on challenging Complex VQA benchmarks, including GQA, A-OKVQA, and OKVQA, after training on VQAv2. Our experimental results demonstrate that MV-CoRe consistently outperforms established LVLM baselines, achieving an overall accuracy of 77.5% on GQA. Ablation studies confirm the critical contribution of both object and scene graph features, and human evaluations further validate MV-CoRe’s superior factual correctness and reasoning depth, underscoring its robust capabilities for deep visual and conceptual understanding. I Introduction Visual Question Answering (VQA) stands as a pivotal task at the intersection of computer vision and natural language processing, requiring machines to comprehend visual content and answer questions posed in natural language [1]. While significant progress has been made in standard VQA benchmarks, a growing emphasis has shifted towards Complex Visual Question Answering (Complex VQA). This advanced variant demands not only factual recall but also sophisticated multi-modal reasoning, including multi-hop inference, common-sense reasoning, and the integration of external knowledge [2]. The ability to accurately answer complex questions about images is crucial for developing truly intelligent systems capable of deep visual and conceptual understanding, paving the way for applications in areas such as intelligent assistants, educational tools, and advanced robotics, including specialized domains like multi-modal medical diagnosis requiring role-specialized collaboration [3]. Despite the remarkable advancements brought forth by large vision-language models (LVLMs) in recent years, their performance on Complex VQA tasks often encounters significant challenges. Many existing LVLMs primarily rely on high-level, global features from pre-trained visual and language encoders, which may prove insufficient for tasks requiring fine-grained understanding of objects, their attributes, and intricate relationships within a scene [4]. While approaches like visual in-context learning for LVLMs [5] offer promising avenues for adapting these models to new tasks with limited examples, these models frequently struggle with questions necessitating external, domain-specific knowledge or complex logical deductions that extend beyond simple visual recognition. This limitation motivates the need for a more robust framework that can deeply integrate diverse forms of visual and linguistic information, moving beyond superficial feature concatenation to facilitate genuine multi-modal conceptual reasoning. To address these challenges, we propose a novel model named MV-CoRe (Multimodal Visual-Conceptual Reasoning). Our approach is designed to enhance the performance of Complex VQA by meticulously fusing three distinct types of input features: global embeddings from pre-trained Vision Large Models (VLMs), contextual embeddings from pre-trained Language Large Models (LLMs), and fine-grained semantic-aware visual features. The semantic-aware visual features are particularly crucial, encompassing object detection characteristics (e.g., bounding boxes and categories from detectors like YOLO or Faster R-CNN) and scene graph representations that explicitly capture relationships between objects in an image. These diverse feature sets are then processed and deeply integrated using an innovative Multimodal Fusion Transformer architecture. This specialized transformer is engineered to enable rich cross-modal attention and interaction, allowing the model to develop a more profound understanding of the image content and the associated linguistic query, thereby facilitating complex reasoning processes. For our experimental evaluation, we primarily train the MV-CoRe model on the comprehensive VQAv2 dataset, which serves as a foundational benchmark for general visual question answering [6]. To rigorously assess its capabilities in Complex VQA scenarios, we then evaluate MV-CoRe’s generalization and reasoning abilities on three challenging target datasets: GQA [7], which focuses on graph-based reasoning over images; A-OKVQA [8], requiring external knowledge for answers; and OKVQA [8], another open-domain knowledge-based VQA dataset. The training strategy involves pre-processing visual and textual data to extract the aforementioned feature embeddings, which are subsequently fed into the MV-CoRe model for end-to-end training. Our experimental results demonstrate the superior performance of MV-CoRe compared to existing LVLM baselines on Complex VQA tasks. For instance, on the GQA dataset, our proposed MV-CoRe model achieves an overall accuracy of 77.5%, significantly outperforming a standard LVLM baseline (e.g., LLaVA) which scores 75.2%. Furthermore, we observe that the incremental inclusion of semantic-aware features progressively improves performance, with ”LVLM + Object Features” reaching 76.5% and ”LVLM + Scene Graph” further improving to 76.8%. This highlights the efficacy of our deep feature fusion strategy and the robust reasoning capabilities of our Multimodal Fusion Transformer. In summary, our main contributions are as follows: • We propose MV-CoRe, a novel architecture specifically designed to enhance performance in Complex VQA tasks by integrating global LVLM embeddings with fine-grained semantic-aware visual features. • We introduce a sophisticated Multimodal Fusion Transformer that effectively fuses diverse visual and linguistic information, enabling deeper cross-modal understanding and complex reasoning. • We demonstrate that MV-CoRe achieves state-of-the-art performance on challenging Complex VQA benchmarks, showcasing its superior ability to handle multi-modal fusion reasoning and leverage cross-domain knowledge. II Related Work II-A Evolution of Visual Question Answering The evolution of Visual Question Answering (VQA) has been comprehensively surveyed by [9], which traces its development from foundational approaches to the profound impact of transformer architectures and vision-language pre-training on recent advancements. This work meticulously details key models, datasets, and techniques that have shaped the VQA landscape, while also addressing current challenges and future trajectories, including the emergence of large multimodal language models and the integration of external knowledge. Complementing this broader view, [10] advances VQA by introducing an answer-driven framework that explicitly models the influence of dialogue responses on visual state estimation, thereby enhancing multi-modal reasoning through answer-informed attention mechanisms. Furthermore, the critical role of regional visual information in knowledge-based VQA is underscored by [11], which demonstrates that improved utilization of object relationships within visual representations significantly boosts performance. Their proposed REVIVE method addresses limitations in existing knowledge-based VQA by effectively leveraging explicit object region information during both knowledge retrieval and the final answering stages, achieving state-of-the-art results on the OK-VQA dataset. Relatedly, advancements in leveraging knowledge graphs for question answering, including efforts to improve zero-shot cross-lingual transfer for multilingual question answering over knowledge graphs, have also contributed to the broader field of complex QA [12]. In a related vein, [13] provides a thorough review of attention mechanisms in VQA, systematically categorizing and analyzing various multimodal fusion strategies for integrating visual and textual information. This review highlights how attention mechanisms empower VQA models to selectively focus on pertinent image regions guided by the question, charting advancements from rudimentary fusion operations to more sophisticated approaches. II-B Large Vision-Language Models and Multimodal Feature Integration The integration of Large Vision-Language Models (LVLMs) within robotic vision tasks is comprehensively surveyed by [14]. This work meticulously examines their advantages over traditional fusion methods, identifies key research challenges such as cross-modal alignment and efficient fusion strategies, and systematically reviews their applications across various robotic vision domains, including semantic scene understanding, SLAM, and navigation, while also proposing future research directions for advancing multimodal perception in robots. Addressing the significant computational burden associated with evaluating large vision-language models, [4] proposes an efficient subset construction methodology based on farthest point sampling, which substantially reduces data requirements while maintaining high correlation with full evaluations, offering a practical solution for comprehensive LVLM assessment. In the broader context of leveraging pretrained features for multimodal tasks, [15] investigates the integration of features from pretrained models for graph learning, proposing a novel method to enhance graph-agnostic node representations by aligning them with graph structure through a metric of feature homophily; while specifically focusing on multimodal large language models as vision learners, the underlying principle of refining pretrained features for downstream applications aligns with the goal of improving multimodal feature integration. However, potential limitations of contrastive learning in Vision-Language Models (VLMs), particularly when handling nuanced multimodal data, are explored by [16], which introduces a framework for synthetic shortcut injection to assess representation learning efficacy, demonstrating that standard contrastive training can lead VLMs to exploit shortcuts rather than learning comprehensive multimodal representations. To quantitatively assess the quality of pre-training for LVLMs, [17] introduces the Modality Integration Rate (MIR) as a novel metric, providing insights into modality alignment without requiring supervised fine-tuning and aiding in the optimization of visual and textual information integration crucial for effective cross-modal attention. Further advancing LVLMs, approaches like visual in-context learning have been explored to enhance their adaptability and performance on new tasks [5]. Addressing a crucial gap, [18] introduces contextual object detection, a novel problem focused on understanding visible objects within human-AI interactive contexts, and proposes ContextDET, a unified multimodal model capable of end-to-end differentiable visual-language context modeling for object localization and identification through a generate-then-detect framework. Beyond specific task improvements, research into weak to strong generalization for large language models with multi-capabilities aims to broaden their applicability and robustness [19]. In the realm of structured multimodal understanding, [20] introduces Action Genome, a novel representation that decomposes actions into spatio-temporal scene graphs, capturing dynamic object interactions and relationships within videos, thereby improving performance and enabling few-shot recognition when integrated into action recognition models. The increasing complexity of human instructions for AI systems has also spurred new benchmarks and agent frameworks for tasks like complex instruction-based image generation [21] and image editing via compositional dependencies [22], highlighting the need for advanced understanding of intricate commands. Furthermore, the application of large vision-language models extends to specialized domains, such as improving medical LVLMs with abnormal-aware feedback [23] and developing modular multi-agent frameworks for multi-modal medical diagnosis through role-specialized collaboration [3]. These efforts underscore the growing demand for highly capable and specialized multimodal AI systems, including those leveraging novel architectures like State Space Models for domain-specific recognition tasks such as insect recognition [24]. Finally, [25] introduces Groma, a novel Multimodal Large Language Model (MLLM) that significantly enhances visual grounding by directly integrating localized visual tokens into its architecture, enabling fine-grained region-level perception and grounding capabilities within the language model itself through embedding localization directly into the image tokenization process. III Method Our proposed MV-CoRe (Multimodal Visual-Conceptual Reasoning) model is meticulously designed to address the complexities inherent in Complex Visual Question Answering (Complex VQA) tasks. The core idea behind MV-CoRe is to leverage and deeply integrate diverse forms of visual and linguistic information, moving beyond superficial feature concatenation to enable sophisticated multi-modal conceptual reasoning. This section details the architecture of MV-CoRe, including its feature extraction mechanisms and the innovative Multimodal Fusion Transformer at its heart. III-A Overall Architecture The MV-CoRe architecture is built upon a multi-stream input design, processing distinct types of information to build a comprehensive understanding of the visual scene and the associated question. It comprises three primary input streams: global visual embeddings derived from large vision models, linguistic embeddings from large language models, and fine-grained semantic-aware visual features. These diverse feature sets are then fed into a specialized Multimodal Fusion Transformer, which is responsible for learning intricate cross-modal interactions and producing a unified representation for answer prediction. The overall process can be conceptualized as fusing high-level global understanding with detailed semantic insights to facilitate robust reasoning, allowing the model to answer complex questions requiring a deep comprehension of both visual content and linguistic nuances. III-B Feature Extraction To capture a rich representation of both the image and the question, MV-CoRe employs a sophisticated feature extraction pipeline that generates three distinct types of embeddings. III-B1 Vision Large Model Embeddings For global visual understanding, we utilize pre-trained Vision Large Models (VLMs), such as Vision Transformer (ViT) or CLIP, to extract comprehensive image embeddings. Given an input image II, a VLM encoder ℰV\mathcal{E}_{V} processes II to yield a fixed-dimensional global visual embedding EV∈ℝdVE_{V}\in\mathbb{R}^{d_{V}}: EV=ℰV​(I)\displaystyle E_{V}=\mathcal{E}_{V}(I) (1) These embeddings encapsulate high-level visual concepts and serve as a strong foundational representation for the entire image, providing broad contextual awareness. III-B2 Language Large Model Embeddings To understand the nuances of the natural language question, we employ pre-trained Language Large Models (LLMs), like LLaMA or models from the GPT series. For an input question QQ, an LLM encoder ℰL\mathcal{E}_{L} generates a sequence of contextualized word embeddings EL∈ℝL×dLE_{L}\in\mathbb{R}^{L\times d_{L}}, where LL is the sequence length and dLd_{L} is the embedding dimension: EL=ℰL​(Q)\displaystyle E_{L}=\mathcal{E}_{L}(Q) (2) These embeddings capture the syntactic and semantic structure of the question, providing rich linguistic context essential for accurate query interpretation. III-B3 Semantic-Aware Visual Features Beyond global visual embeddings, MV-CoRe incorporates fine-grained semantic information crucial for complex reasoning. This includes object-level details and relational knowledge. Object Detection Features: We utilize pre-trained object detection models, such as YOLO or Faster R-CNN, to identify and localize objects within the image. For an input image II, the object detector identifies No​b​jN_{obj} distinct objects. For each detected object ii, its bounding box coordinates Bi=(xi,yi,wi,hi)B_{i}=(x_{i},y_{i},w_{i},h_{i}) and its corresponding class label CiC_{i} are extracted. These are then transformed into feature vectors Fo​b​j,iF_{obj,i} representing individual object characteristics. The collection of all detected object features forms a set Fo​b​j∈ℝNo​b​j×do​b​jF_{obj}\in\mathbb{R}^{N_{obj}\times d_{obj}}, where do​b​jd_{obj} is the feature dimension: Fo​b​j=Detector​(I)\displaystyle F_{obj}=\text{Detector}(I) (3) These features provide specific, localized visual cues, enabling the model to focus on relevant entities within the scene. Scene Graph Features: To capture the relationships between objects, we employ a scene graph generator. A scene graph represents the image content as a graph G=(VG,EG)G=(V_{G},E_{G}), where nodes VGV_{G} correspond to detected objects (with their attributes) and edges EGE_{G} represent the relationships between them (e.g., ”person riding bicycle”). These graph structures are then encoded into a set of feature vectors Fs​g∈ℝNs​g×ds​gF_{sg}\in\mathbb{R}^{N_{sg}\times d_{sg}} that explicitly model object interactions and their attributes: Fs​g=SceneGraphGenerator​(I,Fo​b​j)\displaystyle F_{sg}=\text{SceneGraphGenerator}(I,F_{obj}) (4) Scene graph features are vital for multi-hop reasoning and understanding compositional relationships within the scene, as they provide an explicit representation of how objects relate to each other. III-C Multimodal Fusion Transformer The extracted features—EVE_{V} (global visual), ELE_{L} (linguistic), Fo​b​jF_{obj} (object-centric), and Fs​gF_{sg} (relational)—are then fed into our novel Multimodal Fusion Transformer. This transformer architecture is designed to perform deep, interactive fusion across these diverse modalities. Initially, each feature type is projected into a common latent space of dimension DD to ensure compatibility for fusion. Let PV​(⋅)P_{V}(\cdot), PL​(⋅)P_{L}(\cdot), Po​b​j​(⋅)P_{obj}(\cdot), and Ps​g​(⋅)P_{sg}(\cdot) denote these linear projection layers: HV\displaystyle H_{V} =PV​(EV)\displaystyle=P_{V}(E_{V}) (5) HL\displaystyle H_{L} =PL​(EL)\displaystyle=P_{L}(E_{L}) (6) Ho​b​j\displaystyle H_{obj} =Po​b​j​(Fo​b​j)\displaystyle=P_{obj}(F_{obj}) (7) Hs​g\displaystyle H_{sg} =Ps​g​(Fs​g)\displaystyle=P_{sg}(F_{sg}) (8) These projected features HV,HL,Ho​b​j,Hs​gH_{V},H_{L},H_{obj},H_{sg} serve as inputs to the subsequent transformer layers. The Multimodal Fusion Transformer consists of multiple layers, each incorporating self-attention and cross-attention mechanisms to learn intricate cross-modal interactions. Intra-modal Self-Attention: Within each layer, each modality (or a combination of visual modalities) undergoes self-attention to capture internal dependencies and refine its representation. For example, for the linguistic features, the self-attention mechanism computes updated representations HL′H_{L}^{\prime} by allowing each token to attend to all other tokens within the linguistic sequence. This process can be formulated as: QL\displaystyle Q_{L} =HL​WQL\displaystyle=H_{L}W_{Q}^{L} (9) KL\displaystyle K_{L} =HL​WKL\displaystyle=H_{L}W_{K}^{L} (10) VL\displaystyle V_{L} =HL​WVL\displaystyle=H_{L}W_{V}^{L} (11) HL′\displaystyle H_{L}^{\prime} =Softmax​(QL​KLTdk)​VL\displaystyle=\text{Softmax}\left(\frac{Q_{L}K_{L}^{T}}{\sqrt{d_{k}}}\right)V_{L} (12) where WQL,WKL,WVLW_{Q}^{L},W_{K}^{L},W_{V}^{L} are learnable weight matrices for queries, keys, and values, respectively, and dkd_{k} is the dimension of the keys. Similar self-attention operations are applied to Ho​b​jH_{obj} and Hs​gH_{sg} to capture intra-modal relationships among objects and within the scene graph structure. The global visual feature HVH_{V} may also undergo self-attention if it’s a sequence, or be broadcast if it’s a single vector. Cross-modal Attention: This is the crucial component for fusion, enabling query tokens from one modality to attend to key-value pairs from another, facilitating information exchange. A common cross-attention operation can be formulated as: CrossAttention​(QA,KB,VB)=Softmax​(QA​KBTdk)​VB\displaystyle\text{CrossAttention}(Q_{A},K_{B},V_{B})=\text{Softmax}\left(\frac{Q_{A}K_{B}^{T}}{\sqrt{d_{k}}}\right)V_{B} (13) where QAQ_{A} are queries derived from modality A, and KB,VBK_{B},V_{B} are keys and values derived from modality B. Our transformer employs a series of interleaved cross-attention blocks, allowing deep, iterative interaction between the different modalities. For instance, linguistic queries (QLQ_{L}) can attend to global visual context (KV,VVK_{V},V_{V}), object features (Ko​b​j,Vo​b​jK_{obj},V_{obj}), and scene graph features (Ks​g,Vs​gK_{sg},V_{sg}). Conversely, visual features can query linguistic cues to disambiguate objects or relationships. An example of linguistic features attending to global visual features would be: QL→V\displaystyle Q_{L\to V} =HL​WQL→V\displaystyle=H_{L}W_{Q}^{L\to V} (14) KL→V\displaystyle K_{L\to V} =HV​WKL→V\displaystyle=H_{V}W_{K}^{L\to V} (15) VL→V\displaystyle V_{L\to V} =HV​WVL→V\displaystyle=H_{V}W_{V}^{L\to V} (16) HL,updated\displaystyle H_{L,\text{updated}} =CrossAttention​(QL→V,KL→V,VL→V)\displaystyle=\text{CrossAttention}(Q_{L\to V},K_{L\to V},V_{L\to V}) (17) This iterative interaction across multiple layers facilitates the alignment and fusion of information from different granularities and sources, building a comprehensive multimodal understanding. The outputs of these fusion layers are concatenated to form a unified multimodal representation Hf​u​s​e​dH_{fused}. This final representation is then passed through a multi-layer perceptron (MLP) head to predict the answer to the question. The model maps Hf​u​s​e​dH_{fused} to a probability distribution over the answer vocabulary: P​(Answer|I,Q)=Softmax​(MLP​(Hf​u​s​e​d))\displaystyle P(\text{Answer}|I,Q)=\text{Softmax}(\text{MLP}(H_{fused})) (18) This architecture ensures that the model can leverage both broad contextual understanding from VLMs and LLMs, alongside precise semantic details from object detectors and scene graphs, to perform complex reasoning for tasks like those found in A-OKVQA or OKVQA. III-D Training Objective The MV-CoRe model is trained end-to-end using a standard cross-entropy loss function. For a given image-question pair (I,Q)(I,Q) and its ground-truth answer AA, the model aims to minimize the negative log-likelihood of the predicted answer distribution. If the ground-truth answer is represented as a one-hot vector or a soft-target distribution AA, the loss function is defined as: ℒ=−∑iAi​log⁡(P​(Answeri|I,Q))\displaystyle\mathcal{L}=-\sum_{i}A_{i}\log(P(\text{Answer}_{i}|I,Q)) (19) where AiA_{i} represents the ground-truth probability for answer ii (e.g., 1 for the correct answer in a single-choice setting, or a normalized count for multiple-annotator settings). During training, the parameters of the projection layers, the Multimodal Fusion Transformer, and the final prediction head are optimized. The pre-trained encoders for VLM, LLM, object detection, and scene graph generation are typically kept frozen or fine-tuned with a smaller learning rate. This strategy allows the model to effectively learn how to integrate and reason over the rich features provided by the pre-trained components, while benefiting from their strong initial representations. IV Experiments In this section, we detail the experimental setup, evaluate the performance of our proposed MV-CoRe model against several baselines, present an ablation study to validate the effectiveness of its key components, discuss human evaluation results, and provide qualitative examples along with an error analysis. IV-A Experimental Setup IV-A1 Datasets Our model is primarily trained on the VQAv2 dataset [1], a comprehensive visual question answering benchmark widely used for general VQA tasks. To rigorously assess MV-CoRe’s capabilities in handling complex reasoning and external knowledge, we evaluate its performance on three challenging target Complex VQA datasets: • GQA [7]: Focuses on compositional and multi-hop reasoning over scene graphs. • A-OKVQA [8]: Requires external common-sense and domain-specific knowledge to answer questions. • OKVQA [8]: An open-domain knowledge-based VQA dataset. IV-A2 Implementation Details For Vision Large Model embeddings, we utilize features extracted from a pre-trained ViT-L/14 backbone. For Language Large Model embeddings, we employ a LLaMA-