Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for text-to-video generation that incorporates multimodal large language model (MLLM) feedback into the diffusion sampling process, enabling real-time semantic correction. By implementing a Semantic Assessment Supervisor and a Semantic Modification Assistant, the authors achieve significant improvements in semantic alignment, visual fidelity, and temporal consistency during video synthesis. The method operates without altering model parameters, demonstrating its efficiency and effectiveness across various benchmarks.
Real-time semantic corrections during video generation can drastically enhance visual fidelity and coherence without retraining the model.
Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.
