Search papers, labs, and topics across Lattice.
This paper introduces Semantic Task Completion Video Generation, a novel task that emphasizes both achieving intended outcomes and maintaining semantic grounding in video generation. The authors constructed SemComp-Data, a comprehensive dataset that includes reference images, detailed instructions, and outcome-centric video clips, enabling systematic evaluation of video generation models. Results indicate that current models struggle to achieve intended outcomes while ensuring semantic alignment with reference images, highlighting significant challenges in the field.
Achieving intended outcomes in video generation while maintaining semantic relevance is more challenging than previously thought, with current models falling short.
We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
