Search papers, labs, and topics across Lattice.
DirectorBench is introduced as a personalized multi-agent diagnostic benchmark for evaluating long-form video generation across script, visual, audio, cross-modal, and stability dimensions using 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria. Evaluation of 4 long-form video generation workflows reveals a significant bottleneck in transition quality (averaging 0.256), despite higher prompt-level user demand fulfillment (averaging 0.71). Human evaluation validates DirectorBench's ability to capture human-perceptible quality differences and workflow/profile-dependent failure modes, highlighting the limitations of aggregate scoring.
Long-form video generation struggles with transitions, scoring only 0.256 on transition quality even when prompt fulfillment is high (0.71), revealing a critical bottleneck exposed by the new DirectorBench diagnostic benchmark.
Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinematic control, audio, and cross-modal synchronization. However, evaluating such videos remains challenging, since existing benchmarks largely focus on local visual quality, short-horizon temporal consistency, or generic prompt alignment, and provide limited diagnosis of workflow failures and user-dependent preferences. We introduce DirectorBench, a personalized multi-agent diagnostic benchmark for long-form video generation. DirectorBench evaluates generated videos with respect to 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria across 5 dimensions: script, visual, audio, cross-modal, and stability. Instead of reducing quality to a single aggregate score, DirectorBench localizes checkpoint-level bottlenecks and supports profile-aware evaluation. We evaluate 4 long-form video generation workflows, 6 base LLMs, and 7 user profiles. Across workflows, DirectorBench reveals a between-unit bottleneck: transition quality averages only 0.256 and reaches 0.356 for the best workflow, while prompt-level user demand fulfillment averages 0.71. We further conduct human evaluation with 14 annotators to validate the alignment between DirectorBench and human judgment. The results show that DirectorBench captures human-perceptible quality differences and reveals workflow- and profile-dependent failure modes that are hidden by aggregate scoring. These findings highlight the importance of diagnostic and profile-aware benchmarking for long-form video generation.
