Search papers, labs, and topics across Lattice.
This paper introduces AudioMap, a novel reinforcement learning framework for time-aware dense audio captioning (TDAC) that addresses the limitations of existing methods by utilizing a unified cloze-and-choice reward paradigm. By implementing the Evidence Sufficiency Reward (ESR) and Event-Conditioned Temporal Reward (ECTR), the framework enhances fine-grained accuracy and temporal alignment in audio descriptions. The authors also present the AudioMapCap-44K dataset, enabling extensive evaluation that demonstrates AudioMap's state-of-the-art performance compared to both open-source and proprietary models.
Fine-grained audio captioning just got a major upgrade鈥擜udioMap achieves state-of-the-art results by redefining how we reward temporal accuracy and descriptive richness in audio events.
Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at https://github.com/ryysayhi/AudioMap.
