Search papers, labs, and topics across Lattice.
This paper introduces CAPEval, a novel benchmark that decouples caption evaluation into two distinct metrics: Coverage, which assesses how comprehensively a caption reflects ground-truth visual content, and Precision, which measures the factual correctness of the claims made in the caption. Through experiments involving ten captioning models, the authors demonstrate that Coverage is more closely linked to understanding tasks, while Precision is a better predictor for text-to-image generation tasks. This nuanced evaluation framework provides clearer insights into caption quality and offers guidance for optimizing captioning models based on specific application needs.
Captions are not just text; their quality can be dissected into Coverage and Precision, revealing distinct impacts on multimodal understanding and generation tasks.
Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.
