Search papers, labs, and topics across Lattice.
This paper introduces Goku, a novel dataset comprising 2 million instruction-aligned video editing pairs that expands the scope of video editing tasks from basic appearance changes to complex multi-task and structural manipulations. The authors address the challenges of data synthesis for these intricate tasks by implementing a sophisticated data synthesis pipeline and a progressive filtering system to ensure data reliability. Goku-Edit, the proposed editing model, utilizes a multi-layer language model and a dual-branch architecture, achieving up to an 8% performance improvement in instruction adherence compared to existing models on the new Goku-Bench benchmark.
Goku redefines the landscape of video editing datasets by enabling complex, multi-task editing capabilities that surpass traditional single-task limitations.
Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.
