Search papers, labs, and topics across Lattice.
This paper introduces SciFigAlign, a fine-tuned multimodal scoring system designed to evaluate scientific figures based on their alignment with manuscript evidence, addressing the inadequacies of traditional image quality assessment methods. By leveraging an annotated dataset of 3,857 scientific figures rated on dimensions crucial for peer review, the authors demonstrate that their model significantly outperforms existing LLM-based methods, achieving a 59% relative error reduction in figure quality assessment. Key findings reveal that effective scoring hinges on the integration of visual content with contextual manuscript information, underscoring the necessity of learned alignment in scientific figure evaluation.
SciFigAlign achieves a remarkable 59% reduction in error over traditional LLM-based scoring methods by grounding figure assessments in manuscript context.
Scientific figure assessment in peer review differs fundamentally from general image quality evaluation: a figure must be visually legible, faithfully support the manuscript's claims, and communicate evidence with a clear visual hierarchy. However, if we apply traditional image assessment methods to scientific figure quality assessment, limitations emerge: classic IQA models capture perceptual quality or aesthetics but cannot judge whether a figure serves the paper's scientific argument; CLIP-based methods assess generic image-text correspondence, yet lack understanding of manuscript context; and zero-shot LLM/VLM judges, when repurposed for figure scoring, often yield overly concentrated scores with limited fusion of visual and textual evidence. We introduce an annotated dataset of 3,857 scientific figures from peer-reviewed conference papers, each rated along four peer-review-oriented dimensions: Clarity, Relevance, Informativeness, and Structure. We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence. Given a figure crop, caption, citing paragraphs, and light paper context, SciFigAlign fine-tunes CLIP and SciBERT end-to-end with per-modality cross-attention and CubeMLP fusion, jointly optimizing SmoothL1 regression with a within-paper ranking hinge loss. Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set with n = 396, a 59% relative error reduction over the best LLM-as-judge baseline with MAE 0.864. Ablations confirm that manuscript-grounded inputs, citing-context denoising, and ranking supervision are all critical, showing that scientific figure assessment requires learned alignment between visual content and manuscript evidence rather than prompting alone, even with state-of-the-art VLMs.
MIT CSAIL