Search papers, labs, and topics across Lattice.
This paper explores Inference-Time Scaling (ITS) for joint audio-video generation, addressing the challenges of synthesizing semantically aligned and synchronized outputs without extensive training resources. The authors introduce a multi-verifier framework to overcome limitations of single-objective guidance and identify an optimal combination that enhances quality across multiple dimensions. Their novel Adaptive Reward Weighting (ARW) algorithm effectively aggregates diverse reward signals, leading to significant improvements in semantic alignment, perceptual quality, and synchronization on benchmark datasets.
Achieving robust audio-video generation without extensive training resources, this study reveals that a multi-verifier approach can dramatically enhance output quality across multiple dimensions.
Joint audio-video generation aims to synthesize realistic audio-video pairs that are both semantically aligned with text prompts and precisely synchronized. While existing joint audio-video generation models often require substantial training resources to improve fidelity, Inference-Time Scaling (ITS) has recently emerged as a promising training-free alternative in single-modality domains. However, extending ITS from a single modality to multimodal domains is non-trivial, as it requires balancing multiple heterogeneous objectives. In this paper, we present the first comprehensive study of ITS for joint audio-video generation. We first demonstrate that a multi-verifier framework is essential to address the limitations of single-objective guidance, including asymmetric performance trade-offs and verifier hacking. Through systematic analysis, we then identify an optimal multi-verifier combination that yields balanced improvements across all quality dimensions. Finally, to effectively aggregate diverse reward signals, we propose Adaptive Reward Weighting (ARW), a novel test-time optimization algorithm. ARW treats reward aggregation as an online optimization problem, utilizing learnable parameters to calibrate reward variances without requiring prior knowledge of reward distributions, thereby ensuring robust multi-objective selection. Experimental results on VGGSound and JavisBench-mini benchmarks demonstrate that our framework significantly enhances semantic alignment, perceptual quality, and audio-visual synchronization of generated outputs. Synthesized samples and code are available on the project page: https://jung-jaemin.github.io/ITS-AVGen-Proj.
