Search papers, labs, and topics across Lattice.
The paper introduces Chartographer, a framework for generating counterfactual charts to evaluate the visual reasoning capabilities of Vision-Language Models (VLMs) more rigorously. Chartographer reverse engineers charts into executable code, generates seed-controlled counterfactual variants, and derives new answers, allowing for controlled variation of chart elements while maintaining task structure. Experiments on existing chart QA datasets reveal that VLMs often struggle to generalize to counterfactual charts, particularly when novel visual reasoning pathways are required, even after correctly answering the original chart question.
VLMs that ace standard chart QA benchmarks can still fail spectacularly when presented with subtly altered, counterfactual versions of the same charts.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
