Search papers, labs, and topics across Lattice.
The paper introduces ChartEditBench, a new benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to perform incremental, visually grounded chart editing through multi-turn interactions. This benchmark contains 5,000 difficulty-controlled modification chains and a human-verified subset, addressing the gap in evaluating MLLMs in realistic exploratory data analysis scenarios. Experiments using ChartEditBench on state-of-the-art MLLMs reveal performance degradation in multi-turn settings due to error accumulation and context breakdowns, especially for data-centric transformations.
MLLMs struggle with multi-turn chart editing, forgetting context and accumulating errors, especially when the edits involve data transformations, not just styling.
While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.
