Search papers, labs, and topics across Lattice.
This paper introduces PercepCap, a novel video captioning framework that enhances spatio-temporal understanding by explicitly generating a perception trace before producing captions. By employing a perceive-describe generation chain and a two-stage training strategy that includes Perceive-then-Describe Supervised Fine-tuning and Perception-Grounded Reinforcement Learning, PercepCap effectively aligns perceptual evidence with final captions. The results show that PercepCap outperforms the Qwen3-VL baseline in both direct captioning and caption-to-QA evaluations, indicating significant improvements in caption quality and perceptual accuracy.
Captions generated by PercepCap are not only more accurate but also grounded in explicit spatio-temporal perception, revealing the underlying reasoning behind each description.
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
