Search papers, labs, and topics across Lattice.
The paper introduces OmniRetriever, a 7B parameter audio-video-text encoder trained using a novel "fusion-as-teacher distillation" approach, where a fused AVT embedding acts as a teacher signal for single-modal embeddings. This distillation is paired with a Tuple-InfoNCE loss for direct supervision of the fused embedding. Experiments on six zero-shot retrieval benchmarks and a new AVT retrieval benchmark (OmniRetriever-Bench) demonstrate that OmniRetriever-7B outperforms both closed-source models like Gemini Embedding 2 and existing open-source AVT methods.
OmniRetriever-7B leapfrogs Gemini Embedding 2 in zero-shot audio-video-text retrieval, proving that smart training objectives can close the gap with closed-source giants.
Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.
