Search papers, labs, and topics across Lattice.
The paper introduces ChartAct, a new benchmark for evaluating dynamic chart understanding, which requires models to interact with charts through actions like hovering and clicking to access information. They curated 673 dynamic charts from real websites, creating 1,440 question-answer pairs across two environments (Dynamic Chart and Dashboard Chart). Evaluation of 11 multimodal models revealed limitations in dynamic chart understanding, with the best model (Claude-Opus-4.7) achieving only 84.5% accuracy, highlighting the need for further research in this area.
Even the best multimodal models struggle to understand dynamic charts, achieving only 84.5% accuracy on the new ChartAct benchmark that requires interaction to reveal key information.
Charts are widely used to present complex data for analysis and decision making. Existing chart understanding benchmarks mainly focus on static charts, but real-world charts are often dynamic and interactive. Key information may only appear after actions such as hovering, clicking, zooming, or dragging. Dynamic chart understanding therefore requires models to identify visible content, choose proper interactions, and reason over changing chart states. To evaluate this ability, we propose ChartAct, an interactive benchmark for dynamic chart understanding. ChartAct collects and filters 673 dynamic charts from 8 real chart websites, covers 7 common chart types, and constructs 1,440 high-quality question-answer samples. Each sample is instantiated in two environments, Dynamic Chart and Dashboard Chart, to evaluate dynamic chart understanding under different contexts. Based on ChartAct, we systematically evaluate 11 advanced multimodal models and GUI agents. Experimental results show that existing models still have clear limitations in dynamic chart understanding. The strongest model, Claude-Opus-4.7, achieves an average success rate of 84.5\%, while most models remain below 60\%. We also conduct detailed failure attribution and case analysis. ChartAct provides a new benchmark for studying chart understanding in real interactive environments. Codes at https://github.com/wulin-wulin/OSWorld_Chart
