Search papers, labs, and topics across Lattice.
The paper introduces DIRECT, a hierarchical multi-agent framework that formulates video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP). DIRECT decomposes the problem into three levels: Screenwriter (global structure), Director (editing intent), and Editor (shot sequence editing). Experiments on a new benchmark, Mashup-Bench, show DIRECT significantly outperforms existing methods in visual continuity and auditory alignment, as validated by objective metrics and human evaluation.
Achieve professional-grade video mashups by mimicking a human production pipeline, using hierarchical agents to handle global structure, editing intent, and fine-grained shot selection.
Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT
