Search papers, labs, and topics across Lattice.
The paper introduces FlexSED, an open-vocabulary sound event detection system that addresses the limitations of traditional multi-class SED frameworks by enabling free-text sound queries and improving zero/few-shot learning. FlexSED leverages a pretrained audio SSL model and the CLAP text encoder, incorporating an encoder-decoder architecture and adaptive fusion for continuous training. By using LLMs to refine event query selection for training, FlexSED achieves state-of-the-art performance on AudioSet-Strong and exhibits strong zero-shot and few-shot capabilities.
LLMs can now help you find that *specific* sound you're looking for, even if your training data is incomplete.
Despite recent progress in large-scale sound event detection (SED) systems capable of handling hundreds of sound classes, existing multi-class classification frameworks remain fundamentally limited. They cannot process free-text sound queries, which enable more flexible and user-friendly interaction, and they lack zero-shot capabilities and offer poor few-shot adaptability. Although text-query-based separation methods have been explored, they primarily focus on source separation and are ill-suited for SED tasks that require precise temporal localization and efficient detection across large and diverse sound vocabularies. In this paper, we propose FlexSED, an open-vocabulary sound event detection system. FlexSED builds on a pretrained audio SSL model and the CLAP text encoder, introducing an encoder-decoder composition and an adaptive fusion strategy to enable effective continuous training from pretrained weights. To ensure robust supervision, it also employs large language models (LLMs) to assist in event query selection during training, addressing challenges related to missing labels. As a result, FlexSED achieves superior performance compared to vanilla SED models on AudioSet-Strong, while demonstrating strong zero-shot and few-shot capabilities. We release the code and pretrained models to support future research and applications based on FlexSED.
