Search papers, labs, and topics across Lattice.
This paper introduces LSAR, a novel learned sparse retrieval framework that directly maps audio into a sparse lexical space, circumventing the limitations of traditional ASR-based systems and dense retrievers. By employing cross-modal sparse alignment and multiple complementary branches, LSAR captures spoken content, acoustic context, and logical associations, resulting in an indexable representation that maintains keyword-level interpretability. Experimental results demonstrate that LSAR achieves high-recall candidate retrieval with low latency, making it a promising solution for efficient and interpretable audio retrieval in multimodal applications.
LSAR transforms audio retrieval by mapping audio directly into a sparse lexical space, enabling efficient and interpretable retrieval without the pitfalls of transcription.
As Multimodal Large Language Models (MLLMs) expand the scope of retrieval-augmented generation, recommendation, and multimedia search, audio retrieval is expected to become a dependable retrieval component. Yet existing systems struggle to reconcile lexical precision, non-verbal acoustic evidence, and efficient, transparent retrieval. Current approaches mainly follow two paradigms. Cascaded pipelines transcribe audio with automatic speech recognition (ASR) and retrieve over text. They inherit the strengths of lexical matching, but amplify recognition errors and systematically discard non-verbal cues such as acoustic events and speaking style. Dense retrievers bypass transcription, yet they compress each clip into a single embedding, making relevance difficult to inspect and incurring non-trivial indexing and query-time overhead as collections grow. To address these limitations, we propose LSAR (Learned Sparse Audio Retrieval), the first learned sparse retrieval framework for audio that maps audio directly into a sparse lexical space compatible with inverted-index search. LSAR employs cross-modal sparse alignment and multiple complementary branches to model spoken content, acoustic context, and logic associations, producing an indexable representation with explicit term activations. At inference, retrieval proceeds directly from audio without an ASR transcription stage. Experiments on diverse benchmarks covering speech, captioning, and audio question answering show that LSAR effectively maps continuous audio representations into a sparse lexical space. Acting as a robust first-stage retriever, it enables high-recall candidate retrieval with low latency to significantly narrow the search space, while offering keyword-level interpretability. Overall, LSAR introduces a new retrieval paradigm for audio, establishing an interpretable, index-friendly primitive for real-time multimodal RAG and beyond.
