Search papers, labs, and topics across Lattice.
This paper introduces Unison, a benchmark designed to evaluate unified multimodal models by assessing their joint understanding and generation capabilities through 2,169 high-quality task samples. By incorporating comprehensive dimensions such as internal consistency and mutual enhancement, Unison allows for a more holistic evaluation than traditional isolated assessments. Systematic evaluations reveal significant limitations in current models, pointing to future research directions for improving multimodal systems.
Unified multimodal models may be underperforming due to a lack of synergy between understanding and generation, as revealed by the Unison benchmark.
Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.
