Search papers, labs, and topics across Lattice.
This paper addresses the challenges of long-paragraph detailed audio captioning by introducing Self-Check Captioning (SCC), a framework that integrates audio-grounded question answering as a verification mechanism throughout the captioning process. The authors create the Long-paragraph Audio Caption 50k (LACap-50k), a substantial dataset that includes 50,222 audio-visual clips with extensive captions, and introduce Layer-Curvature Supervised Fine-Tuning (LC-SFT) to enhance model performance by weighting tokens based on intermediate-layer evidence. The results show that SCC achieves state-of-the-art performance on multiple benchmarks, outperforming existing open-source methods and competing effectively with proprietary solutions.
A novel framework that combines audio-grounded verification with a rich dataset leads to significant advancements in long-paragraph audio captioning.
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic speech recognition (ASR) audit. Layer-Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.
