Search papers, labs, and topics across Lattice.
This paper introduces ACE-Cap, a novel framework for long-paragraph fine-grained audio captioning that transforms the traditional passive captioning approach into an active evidence acquisition process. By leveraging multi-turn interactions between a Composer and an Instruct model, ACE-Cap enables the system to identify evidence gaps, query audio for specific information, and synthesize comprehensive captions based on accumulated evidence. The key result shows that this method significantly enhances the quality of audio captions by allowing the model to adaptively decide when to collect information and when to finalize the caption, addressing the limitations of existing one-shot generators.
Transforming audio captioning from a passive task into an adaptive, evidence-driven process could redefine how we approach fine-grained audio understanding.
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.
