Search papers, labs, and topics across Lattice.
The authors investigate whether expensive end-to-end multimodal pre-training is necessary for speech-centric omni models by benchmarking native omni backbones against a modular, training-free alternative. Their framework, Training-Free Omni (TFO), pipes confidence-filtered, timestamped Whisper transcripts directly into frozen VLMs alongside visual tokens without any architectural or alignment tuning. Across 56 benchmarks spanning 21 languages, this zero-training routing approach matches or outperforms native omni models on audio-visual tasks while completely circumventing the degradation of visual grounding, coding, and math reasoning typically caused by native omni training.
Expensive end-to-end omni pre-training degrades foundational reasoning and vision, yet simply routing timestamped ASR transcripts into a frozen VLM matches or beats native omni architectures across 56 benchmarks.
Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.
