Search papers, labs, and topics across Lattice.
This paper investigates why naive multimodal fusion degrades audio reconstruction quality in video-enhanced audio tokenizers, identifying the fusion location, contrastive learning objectives, and feature fusion dimensions as key factors. They find that temporal fusion guided by distinctive features is most effective. Based on these findings, they propose Timing-Aware Pre-Quantization Fusion, which successfully integrates visual information into audio tokenizers while preserving audio reconstruction fidelity and improving downstream task performance.
Fusing video with audio tokenizers doesn't have to trash reconstruction quality: timing-aware fusion *before* quantization unlocks better audio understanding without sacrificing fidelity.
Audio tokenization has emerged as a critical component in end-to-end audio language models, enabling efficient discrete representation learning for both audio understanding and generation tasks. However, existing audio tokenizers face fundamental limitations in understanding tasks due to single-modality constraints, particularly when audio signals contain ambiguous or incomplete information. While incorporating additional modality information can significantly enhance audio understanding, current multimodal fusion approaches invariably degrade reconstruction quality. This degradation is unacceptable for end-to-end audio systems that require high-fidelity audio generation capabilities. In this work, we investigate the root causes of reconstruction quality degradation in video-enhanced audio tokenization and present three key findings. First, the location of fusion within the tokenizer architecture is crucial for preserving reconstruction quality. Second, we show that contrastive learning, though effective in continuous representation fusion, is unsuitable for discrete tokenizers as it fails to enhance downstream task performance. Third, while feature-dimension fusion approaches achieve moderate success, we discover that fusing along the temporal axis -- guided by the concept of distinctive features -- yields significantly better results. Building on these insights, we introduce the Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization, the first approach to successfully integrate visual information into audio tokenizer architectures while preserving reconstruction fidelity. Our approach not only maintains high-fidelity reconstruction but also achieves superior performance on downstream understanding tasks compared with audio-only tokenizers and established multimodal fusion baselines.
