Search papers, labs, and topics across Lattice.
This paper evaluates the cross-domain generalizability of Multimodal Chain-of-Thought (Multimodal-CoT) reasoning by testing a two-stage framework (Zhang et al., 2023) on A-OKVQA, OKVQA, and ChartQA datasets. The study investigates the impact of vision features, rationale quality, and architectural choices on performance. Results show that while vision integration reduces hallucination, CoT reasoning struggles with commonsense reasoning across different question types, highlighting limitations in cross-domain generalization.
Multimodal Chain-of-Thought reasoning, despite excelling in science QA, stumbles on commonsense tasks, revealing critical limitations in cross-domain generalization.
While recent work has extended CoT to multimodal settings, achieving state-of-the-art results on science question answering benchmarks like ScienceQA, the generalizability of these approaches across diverse domains remains underexplored. This work presents a comprehensive analysis of Multimodal Chain-of-Thought (Multimodal-CoT) reasoning, evaluating its effectiveness on the A-OKVQA, OKVQA and ChartQA datasets, which requires broad commonsense and world knowledge beyond scientific reasoning. We implement the two-stage framework proposed by Zhang et al. [3], which separates rationale generation from answer inference and integrates vision features through a gated fusion mechanism with T5-based language models. Through systematic ablation studies, we analyze the contributions of vision features, rationale quality, and architectural choices. Our findings reveal that while vision integration significantly reduces hallucination in rationale generation, the effectiveness of CoT reasoning varies substantially across question types, with commonsense reasoning presenting particular challenges. This work provides practical insights for researchers implementing multimodal reasoning systems and identifies key areas for future improvement in cross-domain generalization.
