Search papers, labs, and topics across Lattice.
This paper introduces ChartWalker, a novel framework designed to benchmark Cross-Chart Retrieval-Augmented Generation (RAG) tasks, addressing the limitations of existing benchmarks that either focus on well-structured tables or generate queries with lexical overlap. By employing a hierarchical knowledge graph construction method tailored to charts and a structure-aware sampling algorithm, ChartWalker enables the creation of semantically coherent, multi-hop reasoning paths that enhance query difficulty and granularity. Evaluations reveal substantial performance gaps across major RAG paradigms, highlighting the benchmark's challenge and its potential to inform future research and system design.
ChartWalker reveals significant performance gaps in cross-chart RAG tasks, challenging the status quo of existing benchmarks and paving the way for more robust multi-modal reasoning.
Cross-Chart Retrieval-Augmented Generation (RAG) is critical for complex multi-modal analytical tasks in scientific, business, and political domains. However, existing benchmarks either focus on tables, which are well-structured and textualized, or generate cross-chart questions by simply extracting key points, which often induces lexical overlap between queries and evidence and yields logically inconsistent reasoning chains. To address this, we introduce ChartWalker, a novel framework for constructing challenging cross-chart RAG tasks. ChartWalker features a hierarchical knowledge graph construction method tailored to charts, which organizes entities and relations by granularity to preserve analytical structure. We then propose a structure-aware sampling algorithm that synthesizes semantically coherent, multi-hop reasoning paths, enabling explicit control over query difficulty and granularity for QA generation. Built with this framework, we release ChartWalker-Bench, a comprehensive benchmark spanning diverse domains and cross-chart query types. Extensive evaluations across major RAG paradigms reveal significant performance gaps, underscoring the benchmark's difficulty and utility. Furthermore, we provide ChartWalker-Agent, an agentic baseline to facilitate analysis and inspire future system design.
Stanford HAI