Search papers, labs, and topics across Lattice.
This paper introduces Auto-AEG, a scalable pipeline designed to enhance Open-Vocabulary Audio Event Grounding by automatically constructing supervision data through a combination of synthesized audio clips and multi-model pseudo-labels. By leveraging reinforcement learning with an interval-aware reward function, the method addresses the critical data scarcity issue in training Large Audio-Language Models (LALMs) for precise temporal localization of sound events. The results demonstrate significant performance improvements on both the DESED SED benchmark and the newly released AEGBench, highlighting the effectiveness of this data-driven approach.
Automatically constructed data can dramatically enhance the temporal localization abilities of audio models, overcoming the limitations of manual annotation.
Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.
Tsinghua AI