Search papers, labs, and topics across Lattice.
CrossEdit introduces a unified omni-modal editing model for images, audio, and video capable of zero-shot cross-modal editing by leveraging instruction-following learned in one modality to transfer to others. The model synthesizes large-scale audio editing pairs with compositional instructions and is fine-tuned on this data, self-supervised AV masked reconstruction, and curated cross-modal tasks to achieve this generalization. Evaluation on the new CrossEditBench benchmark and the AV-FES metric demonstrates improved zero-shot AV movie scene editing performance.
Instruction following learned in one modality transfers zero-shot to cross-modal editing tasks, enabling rich AV movie scene edits without paired training data.
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.
