Search papers, labs, and topics across Lattice.
The paper introduces a data generation pipeline to create high-fidelity training quadruplets for instruction-reference-following video editing, addressing the scarcity of paired training data. This pipeline leverages image generative models to synthesize reference scaffolds from existing video editing pairs, resulting in a large-scale dataset called RefVIE. The authors also propose Kiwi-Edit, a unified editing architecture that combines learnable queries and latent visual features for reference semantic guidance, achieving state-of-the-art performance on the new RefVIE-Bench benchmark.
Instruction-based video editing gets a major upgrade with Kiwi-Edit, which uses synthesized reference images to guide edits with unprecedented precision and control.
Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.
