Search papers, labs, and topics across Lattice.
The paper introduces GENFIG1, a benchmark to evaluate Vision-Language Models (VLMs) on their ability to generate "Figure 1" style visual summaries from scientific papers. The task requires VLMs to understand technical concepts, identify salient points, and synthesize coherent and faithful graphics. Experiments on a curated dataset of deep learning papers show that even state-of-the-art VLMs struggle with this task, highlighting the challenge of scientific visual communication for AI.
Generating a single, effective "Figure 1" from a research paper proves surprisingly difficult for even the best Vision-Language Models, revealing a gap in their ability to couple scientific understanding with visual synthesis.
In many science papers, "Figure 1" serves as the primary visual summary of the core research idea. These figures are visually simple yet conceptually rich, often requiring significant effort and iteration by human authors to get right, highlighting the difficulty of science visual communication. With this intuition, we introduce GENFIG1, a benchmark for generative AI models (e.g., Vision-Language Models). GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Solving GENFIG1 requires more than producing visually appealing graphics: the task entails reasoning for text-to-image generation that couples scientific understanding with visual synthesis. Specifically, models must (i) comprehend and grasp the technical concepts of the paper, (ii) identify the most salient ones, and (iii) design a coherent and aesthetically effective graphic that conveys those concepts visually and is faithful to the input. We curate the benchmark from papers published at top deep-learning conferences, apply stringent quality control, and introduce an automatic evaluation metric that correlates well with expert human judgments. We evaluate a suite of representative models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems. We hope this benchmark serves as a foundation for future progress in multimodal AI.
