Search papers, labs, and topics across Lattice.
This paper introduces AVTok, a unified tokenizer designed to enhance audio-video generation by addressing the limitations of traditional dual-branch methods that separate tokenization for audio and video. By employing a dual-stream transformer architecture with shared encoder-decoder components and modal-specific queries, AVTok efficiently encodes audio-video pairs into a compact one-dimensional representation. Experimental results show that AVTok significantly improves audio-video reconstruction and performs well in various downstream tasks, highlighting its potential for advancing multimodal model development.
AVTok achieves superior audio-video synchronization and reconstruction, setting a new standard for unified multimodal generation.
Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.
