Search papers, labs, and topics across Lattice.
This paper introduces AVOC, a novel framework designed to enhance long-form audio-video understanding in Omni-modal Large Language Models by addressing limitations in context windows and information redundancy. By framing multimodal token compression as a top-$K$ retrieval problem, AVOC employs a learnable token compression module that selects a compact subset of tokens based on relevance, importance, and diversity. Experimental results demonstrate that AVOC outperforms existing models on long-form audio-video benchmarks, achieving significant accuracy improvements while maintaining performance on extended-duration tasks.
AVOC achieves a remarkable 4.9-point accuracy boost over the next best model in long-form audio-video comprehension, redefining efficiency in multimodal understanding.
Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited context windows and severe information redundancy. To address these bottlenecks, we propose AVOC, a framework for long-form audio-video understanding in Omni-modal Large Language Models. AVOC introduces a learnable token compression module between the modality encoders and the LLM backbone. We reframe multimodal token compression as a top-$K$ retrieval problem: given a fixed context budget, the module must retrieve a compact subset of tokens that best supports answering the user query. We draw inspiration from three classical Information Retrieval criteria for selecting informative units from a large candidate pool: relevance, importance, and diversity. AVOC instantiates each criterion as a tailored mechanism for audio-video understanding, and integrates them into a unified retrieval-style compression pipeline. Experiments show that AVOC achieves state-of-the-art performance on long-form audio-video benchmarks, surpassing the second-best model by 4.9 and 5.5 points in average accuracy on OmniVideoBench and LVOmniBench, respectively. Moreover, AVOC maintains robust performance on Audio-Video Needle-in-a-Haystack task at durations up to one hour.
