Search papers, labs, and topics across Lattice.
The paper introduces UGC-AVQA, a new benchmark designed to isolate audio-visual association in user-generated content by ensuring questions require both modalities for accurate answering. To improve efficiency in multimodal LLMs, they propose OMAC, a training-free compression method that selectively preserves salient visual memory and temporally grounded audio anchors. They then introduce O-MARC, a compression distillation framework that trains compact models to be robust to these compressed multimodal contexts, achieving superior performance and efficiency compared to full token inference and existing compression methods.
Compressing multimodal LLM inputs with learned memory-augmented distillation not only speeds up inference by 1.5x, but also *improves* accuracy on audio-visual question answering.
Omnimodal large language models enable unified audio video understanding, but long joint token sequences make inference costly, and existing benchmarks do not fully isolate audio visual association in noisy user generated videos. We introduce UGC-AVQA, a public UGC benchmark with 1,000 videos and 4,816 QA pairs, where an audio removal test ensures that benchmark questions require both acoustic and visual evidence. To reduce inference cost, we propose OMAC, a training free plug in compression method that preserves salient visual memory and temporally grounded audio anchors. To further make compact models robust to compressed inputs, we introduce O-MARC, a compression distillation framework for learning with memory compressed multimodal contexts. On Qwen2.5-Omni-3B, O-MARC improves the average score across four benchmarks to 45.8, outperforming full token inference at 44.1 and OmniZip at 41.0. OMAC also keeps inference efficient, reducing latency by 34.6\% (1.53$\times$ speedup) and memory by 34.7\% compared with full token inference.
