Search papers, labs, and topics across Lattice.
This paper introduces Rigel, a novel automatic evaluation metric for image and video captioning that utilizes self-distilled score adaptation to better align with human judgments. By employing a task-specific scoring head distilled from a frozen large language model (LLM) and refining the LLM with human judgment data, Rigel effectively addresses the limitations of existing metrics that rely on large-vocabulary token sets. Experimental results demonstrate that Rigel significantly outperforms state-of-the-art metrics, achieving over 10-point improvements on the ActivityNet-Fact benchmark in a reference-free setting.
Rigel achieves over 10-point improvements in image and video captioning evaluation by aligning automatic metrics with human judgment without relying on large vocabulary sets.
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set. To address this, we propose Rigel, an automatic evaluation metric for image and video captioning, based on self-distilled score adaptation. The metric employs an evaluation-specific scoring head distilled from a frozen LLM, which captures judgment signals in a task-aligned space without relying on large-vocabulary token sets. We then refine the LLM backbone with human judgment data. To train Rigel, we constructed the Vid-Lepus dataset, which contains 3,338 video clips, 33,380 reference captions, and 5,637 candidate captions. Experiments on multiple benchmarks show that Rigel outperforms state-of-the-art metrics, achieving over 10-point improvements on ActivityNet-Fact in the reference-free setting.
