Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce EpiCurveBench, a new benchmark for evaluating VLMs on epidemic curve digitization, consisting of 1,000 real-world epidemic curve images. They also propose EpiCurveSimilarity (ECS), a novel evaluation metric that uses dynamic programming to align predicted and ground-truth time series, addressing the limitations of existing metrics that treat extracted points as unordered key-value pairs. Experiments with six models reveal that even the strongest VLM achieves only 52.3% ECS, highlighting the benchmark's difficulty and the limitations of current VLMs in capturing temporal structure.
Current VLMs struggle to extract meaningful time-series data from charts, achieving only 52% accuracy on a new benchmark of real-world epidemic curves, despite near-perfect scores on existing benchmarks.
Chart-to-data extraction with vision-language models (VLMs) is increasingly evaluated on benchmarks that show diminishing headroom (frontier VLMs exceed 89% on ChartQA) and with metrics that treat extracted points as unordered key-value pairs, ignoring the temporal structure of time series and penalizing small alignment shifts as catastrophic failures. We address both gaps with EpiCurveBench, a benchmark of 1,000 real-world epidemic curve images curated from diverse public-health sources, and EpiCurveSimilarity (ECS), an evaluation metric that aligns predicted and ground-truth series via dynamic programming, tolerating local temporal shifts and gaps while penalizing them proportionally. Evaluating six methods--three frontier closed VLMs, one open VLM, and two specialized chart-extraction systems--we find the strongest model reaches only 52.3% ECS, and that ECS spreads the four general-purpose VLMs over a 25-point range where key-value metrics (RMS, SCRM) compress them into a 5-point band. We further validate ECS against four downstream epidemiological summary statistics, finding that higher ECS predicts smaller errors in total counts, peak timing, and peak magnitude, and higher growth-rate fidelity; across all four, ECS correlates 1.5--3.6 times more strongly than Dynamic Time Warping, which lacks a gap penalty and therefore cannot distinguish a truncated prediction from a temporally faithful one. EpiCurveBench targets a high-impact public-health application--unlocking decades of outbreak data trapped in published figures--but the benchmark and metric apply directly to any structured time-series chart-extraction setting.