Search papers, labs, and topics across Lattice.
This paper introduces AVSCap, a novel framework for omni-modal video captioning that emphasizes explicit cross-modal event binding between visual and audio streams. By constructing the AVSCap-130K training corpus and employing a two-stage training strategy that combines supervised fine-tuning with reinforcement learning, the authors achieve significant improvements in audio-visual synergy and completeness. The results demonstrate that AVSCap-7B outperforms existing models in capturing non-speech audio and effectively linking it to visual events, as validated by the newly developed AVSCapBench benchmark.
AVSCap-7B achieves superior audio-visual synergy, outperforming existing models by effectively linking non-speech sounds to visual actions.
Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence before composing grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities, while sample-efficient reinforcement learning uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than increasing SFT data. Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.
