Search papers, labs, and topics across Lattice.
This paper introduces the Fusion Embedding family, which integrates audio into a unified embedding space originally designed for text, images, and video, leveraging a frozen vision-language backbone. By training a lightweight connector and modality-gated adapters, the authors achieve effective audio-text and audio-image retrieval without requiring paired audio-visual training data. The results show that this approach not only enhances multimodal retrieval capabilities but also maintains efficiency, with both generations training in hours on a single GPU.
Aligning audio with a frozen vision-language model enables seamless audio-image retrieval without any paired training data, revolutionizing multimodal search capabilities.
A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.
