Search papers, labs, and topics across Lattice.
This paper introduces CoVA-SFT, a large-scale dataset designed to enhance multimodal reasoning in language models by integrating visual and textual information through structured reasoning steps. The dataset comprises 51.9K samples with over 222K reasoning steps across various tasks, enabling models to maintain internal visual workspaces while solving problems. Results show that models fine-tuned on CoVA-SFT achieve over double the performance of existing interleaved chain-of-thought baselines on the CoVA-Bench benchmark, although they still lag behind text-only chain-of-thought approaches, indicating areas for further improvement.
Models trained on CoVA-SFT outperform existing multimodal reasoning baselines by over 2x, revealing the potential of structured visual-textual integration in AI.
Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs) by allowing them to decompose problems into intermediate steps. While CoT is widely effective for linguistic tasks, text-only CoT forces models to serialize visual problems into awkward prose. Although architectural solutions exist to process visual inputs, the community lacks a massive, multi-step, self-corrected dataset to teach models how to build and maintain internal visual workspaces when solving purely textual reasoning problems. To address this limitation, we introduce CoVA-SFT, a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks, and CoVA-Bench, a companion benchmark of 1,700 held-out test samples spanning the same tasks for reproducible evaluation. By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions. We validate the dataset by demonstrating that models fine-tuned on CoVA-SFT outperform all interleaved CoT baselines by more than 2x on average on CoVA-Bench, though they still fall short of strong text-only CoT baselines, highlighting open challenges for future work.
BAIR