Search papers, labs, and topics across Lattice.
This paper introduces CineCap, a novel framework for cinematographic video captioning that integrates structured reasoning with spatio-temporal anchors and reinforcement learning to enhance caption comprehensiveness and accuracy. By grounding professional film concepts in explicit visual evidence, CineCap generates detailed and precise descriptions of video content, addressing the challenges of inferring subtle cinematographic elements. Experimental results demonstrate that CineCap outperforms existing state-of-the-art models, establishing a new benchmark in the field with a newly created dataset of 472 annotated video-caption pairs for evaluation.
CineCap achieves a new state of the art in cinematographic video captioning by effectively balancing descriptive completeness with factual accuracy through innovative structured reasoning techniques.
Cinematographic captioning aims to describe how a video is filmed using professional film-language concepts such as camera movement, shot size, depth of field, composition, and shooting angle. This capability is important for fine-grained video understanding and controllable movie-quality video generation, yet remains underexplored in existing multimodal large language models. Unlike question-answering-based evaluation of cinematic understanding, cinematographic captioning requires a unified open-form description over multiple cinematographic dimensions. This task is challenging for two main reasons: the model must infer professional cinematographic concepts from subtle visual evidence, and it must generate captions that are both comprehensive and accurate. Accordingly, we propose CineCap, a framework that combines structured reasoning with spatio-temporal anchors and reinforcement learning with comprehensiveness, accuracy, and gated coverage rewards. The former grounds professional cinematographic descriptions in explicit visual evidence and organizes them into compact atomic reasoning for supervised fine-tuning, while the latter improves the balance between descriptive completeness and factual correctness. In addition, we construct CineCap Bench, a benchmark of 472 manually annotated video-caption pairs for systematic evaluation. Extensive experiments show that CineCap consistently outperforms strong proprietary and open-source baselines, establishing a new state of the art for cinematographic captioning. The code, model checkpoint, and benchmark are publicly available in https://github.com/Hectormxy/CineCap.git.
NUS