Jiehao Chen

×10−61\times 10^{-6} for 3 epochs. We use a batch size of 64. Evaluation metric is overall accuracy for all datasets. IV-B Baselines To demonstrate the efficacy of MV-CoRe, we compare it against several representative baselines: • LVLM Baseline: Represents a typical large vision-language model, such as LLaVA [4], which primarily fuses global visual features from a VLM and linguistic features from an LLM without explicit fine-grained semantic-aware visual features like object detection or scene graph information. • LVLM + Object Features: An extension of the LVLM Baseline that additionally incorporates features from object detectors. This model concatenates or pools object-level features with the global LVLM embeddings before feeding them into a fusion module. • LVLM + Scene Graph: Another extension that integrates scene graph features with the LVLM Baseline. This typically involves encoding scene graph information (e.g., using a graph neural network) and then fusing it with global LVLM embeddings. These baselines allow us to progressively evaluate the impact of incorporating increasingly richer semantic-aware visual information. IV-C Performance Comparison Table I presents the overall accuracy of MV-CoRe and the baseline models on the GQA dataset, which is a strong indicator of multi-hop reasoning capabilities. TABLE I: Overall Accuracy on the GQA Dataset (%) Model Architecture Accuracy (Overall) LVLM Baseline (e.g., LLaVA) 75.2 LVLM + Object Features 76.5 LVLM + Scene Graph 76.8 MV-CoRe (Ours) 77.5 As shown in Table I, our proposed MV-CoRe model consistently outperforms all baseline methods on the GQA dataset, achieving an overall accuracy of 77.5%. This significant improvement demonstrates the effectiveness of our approach in leveraging diverse multimodal features for complex visual reasoning. The incremental gains observed from adding object features and scene graph features to the LVLM baseline further validate the importance of fine-grained semantic-aware visual information for Complex VQA. To further demonstrate MV-CoRe’s proficiency in tasks requiring external knowledge and common-sense reasoning, we present the results on the A-OKVQA and OKVQA datasets in Table II and Table III, respectively. TABLE II: Overall Accuracy on the A-OKVQA Dataset (%) Model Architecture Accuracy (Overall) LVLM Baseline (e.g., LLaVA) 50.1 LVLM + Object Features 52.5 LVLM + Scene Graph 53.0 MV-CoRe (Ours) 55.8 TABLE III: Overall Accuracy on the OKVQA Dataset (%) Model Architecture Accuracy (Overall) LVLM Baseline (e.g., LLaVA) 40.5 LVLM + Object Features 42.1 LVLM + Scene Graph 43.0 MV-CoRe (Ours) 45.2 The results on A-OKVQA (Table II) and OKVQA (Table III) consistently show MV-CoRe outperforming all baselines. For A-OKVQA, MV-CoRe achieves 55.8% accuracy, significantly higher than the LVLM Baseline’s 50.1%. Similarly, on OKVQA, our model reaches 45.2% accuracy, surpassing the 40.5% of the LVLM Baseline. These results underscore MV-CoRe’s superior ability to ground abstract knowledge with fine-grained visual details and relationships, which is crucial for answering questions that necessitate external information. The consistent performance across all three complex VQA datasets (GQA, A-OKVQA, OKVQA) validates the robustness and generalizability of our multimodal fusion approach. IV-D Ablation Study To understand the individual contributions of the semantic-aware visual features and the overall fusion strategy within MV-CoRe, we conduct a detailed ablation study. We evaluate different configurations of our model on the A-OKVQA dataset, as it presents a significant challenge requiring both visual and external knowledge reasoning. The results are summarized in Table IV. TABLE IV: Ablation Study on A-OKVQA Dataset (% Accuracy) Model Configuration Accuracy (Overall) MV-CoRe (Full Model) 55.8 MV-CoRe w/o Object Features 54.0 MV-CoRe w/o Scene Graph Features 53.5 MV-CoRe w/o Semantic-Aware Visual Features (Obj. & SG) 51.2 LVLM Baseline (for reference) 50.1 From Table IV, several key insights can be drawn: • The full MV-CoRe model achieves the highest accuracy of 55.8%, confirming that the synergistic combination of all proposed features is most effective. • Removing Object Features leads to a notable drop in accuracy from 55.8% to 54.0%. This indicates that precise, localized object information is vital for grounding questions and identifying specific entities, particularly in knowledge-based VQA where object attributes often link to external knowledge. • Similarly, excluding Scene Graph Features results in a performance decrease to 53.5%. This highlights the critical role of relational information in understanding complex visual scenes and performing multi-hop reasoning, which is often implicitly required even in knowledge-based questions. • When both Object Features and Scene Graph Features are removed (MV-CoRe w/o Semantic-Aware Visual Features), the accuracy drops significantly to 51.2%. This configuration effectively represents a version of MV-CoRe that relies only on global VLM and LLM embeddings processed by our Multimodal Fusion Transformer. While still outperforming the generic LVLM Baseline (50.1%), this shows that our specialized fusion transformer alone offers some advantage, but the primary gains come from the integration of the semantic-aware visual features. This ablation study unequivocally demonstrates that both object-level and scene graph features contribute substantially to MV-CoRe’s performance, and their combined effect, deeply fused by the Multimodal Fusion Transformer, is crucial for achieving superior results in Complex VQA. IV-E Analysis of Performance The consistent superior performance of MV-CoRe across GQA, A-OKVQA, and OKVQA datasets highlights its effectiveness in tackling Complex VQA. The key to this success lies in its ability to integrate and reason over diverse granularities of visual and linguistic information. • Synergistic Feature Integration: Unlike baselines that primarily rely on global visual features or simple concatenations, MV-CoRe’s Multimodal Fusion Transformer enables deep, interactive fusion. This allows the model to align linguistic queries with precise visual entities (via object features) and their relationships (via scene graph features), while simultaneously leveraging broad contextual understanding from global VLM embeddings. • Enhanced Visual Grounding: The explicit inclusion of object detection features provides strong visual grounding, allowing the model to pinpoint specific entities mentioned in the question or implied by the context. This is particularly beneficial for questions requiring fine-grained visual details or object counting. • Improved Relational Reasoning: Scene graph features offer a structured representation of visual relationships, which is crucial for answering compositional questions (e.g., ”What is the person behind the car doing?”) and multi-hop reasoning (e.g., ”Is the object on the table to the left of the red book?”). This structured understanding enables more robust inference compared to models relying solely on implicit spatial relationships learned from raw image features. • Robustness to Knowledge-Intensive Queries: For datasets like A-OKVQA and OKVQA, which demand external knowledge, MV-CoRe’s ability to link linguistic concepts to specific visual cues (objects, relations) facilitates more accurate knowledge retrieval and application. For instance, if a question asks about the function of an object, identifying that object precisely via object features allows the LLM component to retrieve relevant knowledge more effectively. The observed incremental improvements from adding object and scene graph features to the LVLM baselines, and the significant drop in performance in our ablation study when these features are removed, empirically confirm their indispensable role. The Multimodal Fusion Transformer’s design, with its interleaved self- and cross-attention mechanisms, is central to effectively weaving these disparate information streams into a coherent multimodal representation, enabling the sophisticated reasoning required for Complex VQA. IV-F Human Evaluation To complement the quantitative metrics, we also performed a human evaluation to assess the qualitative aspects of MV-CoRe’s answers, particularly concerning reasoning depth and factual correctness for complex questions. A randomly selected subset of 100 challenging questions from the A-OKVQA test set was presented to three human annotators. They were asked to rate the answers generated by MV-CoRe and the LVLM Baseline on a scale of 1 to 5 for ”Factual Correctness” (1: incorrect, 5: perfectly correct) and ”Reasoning Depth” (1: no reasoning, 5: deep, logical reasoning). The average scores are presented in Table V. TABLE V: Human Evaluation Results on A-OKVQA Subset (Average Score) Model Factual Correctness Reasoning Depth LVLM Baseline 3.8 3.2 MV-CoRe (Ours) 4.3 4.0 The human evaluation results in Table V indicate that MV-CoRe significantly outperforms the LVLM Baseline in both factual correctness and reasoning depth. Human annotators found MV-CoRe’s answers to be more accurate and, crucially, to demonstrate a deeper understanding of the underlying visual content and required external knowledge, aligning with our objective of enhancing complex reasoning capabilities. This qualitative assessment further supports the quantitative improvements observed in Section IV-C. IV-G Qualitative Examples and Error Analysis To provide a deeper insight into MV-CoRe’s reasoning capabilities, we present a selection of qualitative examples from the A-OKVQA dataset. These examples highlight scenarios where MV-CoRe demonstrates superior understanding and reasoning compared to the LVLM Baseline. TABLE VI: Qualitative Examples from A-OKVQA Image Question LVLM Baseline Answer MV-CoRe Answer Example 1