Search papers, labs, and topics across Lattice.
This paper introduces RefCaptioner, a novel two-stage post-training framework for multi-reference image-grounded video captioning, which enhances the generation of factual video descriptions by grounding local visual elements to multiple reference images. The method employs mixed-data supervised fine-tuning (SFT) alongside Hierarchical Coverage-Discounted GRPO to improve reference selection, phrase-level binding, and cross-reference consistency, while maintaining the general capabilities of video captioning. Experimental results demonstrate that RefCaptioner outperforms existing open-source models and is favored in human evaluations for its ability to produce more source-faithful captions for video reconstruction.
RefCaptioner not only outperforms existing models in video captioning but also enables precise grounding of visual elements to multiple reference images, enhancing factual accuracy.
Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.
