Search papers, labs, and topics across Lattice.
This paper introduces ReBind, a novel framework for multi-reference video editing that incorporates semantic instructions with embedded reference tokens to enhance the coordination of information from multiple visual sources. By addressing the critical deficiency of existing methods that fail to establish explicit reference relationships, ReBind-Instruct and ReBind-Edit enable precise bindings between visual attributes and their sources, significantly improving instruction quality. Experimental results show that ReBind outperforms general-purpose multimodal large language models and achieves state-of-the-art performance in reference image-conditioned video editing tasks.
Embedding reference tokens at semantic positions allows for unprecedented precision in multi-reference video editing, setting a new benchmark for instruction quality.
Recent diffusion-based video generation models have made significant progress in multi-reference image-conditioned video editing. However, existing methods still struggle to coordinate information from multiple visual sources accurately. We identify a critical deficiency in existing approaches. Existing editing instructions lack explicit reference relationships, and most multimodal large language models (MLLMs) cannot generate them reliably. To address this problem, we propose ReBind, a systematic framework that introduces semantic instructions with embedded reference tokens as the intermediate representation for multi-reference image-conditioned video editing. Our key insight is embedding reference tokens at semantic positions to eliminate ambiguity and establish precise bindings between visual attributes and their sources. We develop ReBind-Instruct, a specialized MLLM that learns to establish explicit bindings between visual attributes and their reference sources through a two-stage progressive scheme for precise reference relationships. We further develop ReBind-Edit, which enables lightweight adaptation of text-to-video models to coordinate multiple references by binding visual attributes to their designated sources. Extensive experiments demonstrate that ReBind substantially outperforms general-purpose MLLMs in instruction quality and achieves state-of-the-art performance among open-source methods on reference image conditioned video editing. Our project webpage: https://rebind-mrv2v.github.io/.
