Search papers, labs, and topics across Lattice.
This paper critiques the conventional approach to token compression in OmniLLMs, which typically employs a unified saliency-ranking method that inadvertently favors audio tokens over video. The authors introduce Macer, a training-free compressor that allocates explicit budgets for audio and video before conducting modality-specific ranking, leading to a more equitable distribution of retained capacity. The results show that Macer achieves significant reductions in token cost while maintaining high accuracy, preserving 98.7% of performance at 25% retention on Qwen2.5-Omni-7B and outperforming existing methods by up to 12.9 points on OmniVinci-9B.
Allocating resources before ranking can boost multimodal model efficiency, achieving near-optimal performance with fewer tokens.
Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.
