Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a self-ensembling method to improve chart-to-table extraction using vision-language models (VLMs). The approach samples multiple tabular outputs from a VLM for a given chart image and aggregates them at the cell level using median values to produce a consensus table. They also introduce a new, more complex benchmark, WB-ChartExtract, and demonstrate that self-ensembling improves extraction accuracy by up to 23% on this benchmark compared to single-pass VLM outputs.
Extracting data from charts just got a whole lot better: a simple self-ensembling technique boosts VLM accuracy by up to 23% on complex charts.
Charts effectively convey quantitative information, but the underlying data are often locked in image form, hindering reuse and analysis. Manually digitizing charts is time-consuming and error-prone, motivating automatic chart-to-table extraction. Recent approaches use specialized vision-language models (VLMs), yet performance still lags on charts with many datapoints or substantial stylistic variation. We propose a VLM self-ensembling method that repeatedly samples multiple tabular outputs from the same VLM for a fixed chart image and aggregates them at the level of individual table cells. We align candidate tables and take per-cell medians over numerical values to produce a more accurate consensus table. Our method also includes convergence detection to stop sampling once the aggregated table stabilizes, and uncertainty estimation based on dispersion across samples to help users assess extraction reliability. Because existing chart extraction benchmarks contain relatively simple plots with limited room for improvement, we introduce WB-ChartExtract, a new benchmark built from World Bank data with more complex and stylistically diverse charts; on average, its charts contain 7 times more datapoints than those in the ChartQA benchmark. Across both ChartQA and WB-ChartExtract, our approach improves extraction accuracy over single-pass VLM outputs, yielding up to 23% relative improvement on WB-ChartExtract after ensembling. More broadly, our method helps unlock tabular data previously siloed in chart images, enabling downstream analysis and reuse.