Search papers, labs, and topics across Lattice.
This paper introduces ProCap, a prominence-aware iterative rectification framework that enhances video captioning quality without retraining large vision-language models. By employing a lightweight scoring mechanism to rank detected objects based on their spatial saliency, temporal persistence, and relational dynamics, the method iteratively refines captions to incorporate contextually relevant objects. The results demonstrate a significant increase in perceived completeness (up to 48%) and a reduction in hallucination (up to 45%) compared to a strong pretrained baseline, showcasing its effectiveness in producing more trustworthy video captions.
Captions can be made 48% more complete and hallucination reduced by 45% without retraining models, thanks to a novel prominence-aware rectification approach.
Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected objects to curb hallucination, but apply only a single, fixed correction pass without prioritizing which objects matter most, leaving semantically significant content omitted. We propose a prominence-aware, iterative post-hoc rectification framework that overcomes both limitations without modifying the underlying captioning model's parameters: a lightweight scoring mechanism ranks detected objects by spatial saliency, temporal persistence, and relational dynamics, and an iterative, prompt-driven refinement loop uses this ranking to progressively inject missing yet contextually relevant objects into the caption over multiple rounds. We validate the framework on MSVD and MSR-VTT using object-grounded automatic metrics, a 110-participant human study, and qualitative comparison against ChatGPT and Gemini; in human evaluation, the framework raises perceived completeness by up to 48% and reduces hallucination by up to 45% relative to a strong pretrained captioning baseline, all without retraining or reference captions. These results position prominence-guided iterative rectification as a lightweight, scalable, and model-agnostic route to more complete and trustworthy video captioning, with direct relevance to accessibility, retrieval, and other multimedia understanding applications.
