Search papers, labs, and topics across Lattice.
This paper critiques the conventional approach to chart-to-code generation, which assumes that the gold code is fully observable, and highlights the limitations of this assumption due to the presence of latent raw variables in chart designs. The authors introduce Observation-Aligned supervision, a framework that aligns training targets with identifiable quantities derived from visual observations, such as box statistics and wedge percentages. Experiments on multiple vision-language models demonstrate that this approach significantly enhances the accuracy of observable value recovery, suggesting that effective supervision is crucial for improving chart-to-code generation performance.
Observation-Aligned supervision reveals that traditional chart-to-code training often leads to hallucinations, and aligning targets with identifiable quantities can dramatically improve model performance.
Chart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. For example, a boxplot exposes summary statistics rather than original samples, a pie chart reveals proportions rather than arbitrary raw values, and a histogram shows bin-level mass rather than individual observations. Supervising models to reproduce such non-identifiable quantities encourages hallucination and over-specified code generation. We introduce Observation-Aligned supervision, a rewriting framework that replaces latent raw-data targets with quantities constrained by the visual observation: box statistics for boxplots, wedge percentages for pie charts, and bin weights for histograms. Applying this framework to chart-to-code training data from two sources, we obtain the Observation-Aligned supervision target data. Experiments across multiple VLMs on ChartMimic and ChartX demonstrate consistent improvements in observable value recovery, including under both-executable evaluation. Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives or algorithms, but also supervision targets that respect what is identifiable from the chart image.
