Search papers, labs, and topics across Lattice.
This paper introduces OmniEdit-Bench, a structured benchmark designed to evaluate instruction-based video editing (IVE) by addressing the limitations of existing benchmarks that primarily focus on image editing. By decomposing editing tasks into video-specific dimensions and incorporating a novel accuracy-aware penalty mechanism, the benchmark ensures a more rigorous assessment of editing quality across multiple criteria, including accuracy, preservation, realism, and consistency. Experimental results reveal that current IVE models significantly underperform, highlighting the need for improved methodologies in this emerging field.
Current instruction-based video editing models are far from satisfactory, revealing critical gaps in evaluation that could reshape the field.
Instruction-based video editing (IVE) is an emerging field with broad applications, yet evaluating editing models remains challenging. Existing benchmarks suffer from two major limitations: limited task coverage inherited from image editing, which overlooks video-specific dimensions, and inadequate metrics that fail to measure instruction fidelity, allowing incorrect edits to receive high scores due to strong visual priors from the original video. To address these issues, we introduce a comprehensive and structured benchmark for IVE. Our benchmark decomposes editing tasks into multiple video-specific dimensions, including spatial, temporal, audio, and reference-based editing, extending beyond conventional frame-level evaluation. It also distinguishes explicit and implicit instructions and incorporates reasoning-based scenarios to better reflect real-world requirements. Furthermore, we propose an evaluation framework that assesses editing quality from four complementary dimensions: accuracy, preservation, realism, and consistency, using both human judgments and state-of-the-art vision-language models. To emphasize instruction fidelity, we introduce an accuracy-aware penalty mechanism that conditions other scores on accuracy, preventing visually plausible but incorrect edits from receiving inflated evaluations. Extensive experiments on representative open-source and commercial models show that current IVE models remain far from satisfactory. OmniEdit-Bench provides a comprehensive and reliable testbed for evaluating instruction-based video editing and offers insights into future research directions.
DAMO