Search papers, labs, and topics across Lattice.
The authors introduce MTAVG-Bench 2.0, a benchmark designed to diagnose failures in cinematic expressiveness for multi-talker audio-video generation models, moving beyond basic metrics like lip-sync to assess higher-level cinematic qualities. The benchmark includes a taxonomy of failures across acting, narrative, atmosphere, and audio-visual language, and uses over 10,000 question-answering instances to evaluate models. Experiments reveal that even state-of-the-art omni models like Gemini struggle with complex failures identified by the benchmark, highlighting its utility for systematic failure diagnosis.
Even Gemini struggles with cinematic expressiveness, revealing that current multi-talker audio-video generation models still have a long way to go in capturing the nuances of acting, narrative, and atmosphere.
In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-visual alignment. However, these metrics remain insufficient for assessing cinematic expressiveness in scene-level generation. In multi-character scenes, generation models must go beyond audio-visual realism to convey coherent character performance and other higher-level cinematic qualities. To fill this gap, we introduce MTAVG-Bench 2.0, a benchmark for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation. Unlike prior settings that mainly focus on the quality of basic multi-turn dialogue, MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning acting, narrative, atmosphere, and audio-visual language. Based on this taxonomy, we construct more than 10,000 question-answering evaluation instances, together with subsets for short-drama-level assessment and temporal localization of failure modes, to systematically evaluate the ability of omni large language models to diagnose high-level audio-visual failures. Experimental results show that commercial omni models such as Gemini substantially outperform other evaluators, yet even the strongest models continue to struggle with complex failures in our benchmark. These results demonstrate that MTAVG-Bench 2.0 provides a systematic benchmark for failure diagnosis in cinematic multi-talker audio-video generation.
