Search papers, labs, and topics across Lattice.
Google ResearchA*STARMax PlanckSaarbrücken Research Center for Visual ComputingSUTDMar 6, 2026arXiv:2603.06408The paper introduces Physical Simulator In-the-loop Video Generation (PSIVG), a framework that integrates a physical simulator into the video diffusion process to improve physical realism. PSIVG reconstructs 4D scenes and object meshes from a diffusion-generated template video, uses a physical simulator to generate physically consistent trajectories, and guides the video generator with these trajectories. A Test-Time Texture Consistency Optimization (TTCO) technique further enhances texture consistency by adapting text and feature embeddings based on pixel correspondences from the simulator.
AI-generated videos can now respect physics, thanks to a framework that uses a physical simulator to guide diffusion models, resulting in more realistic and coherent motion.
Recent advances in diffusion-based video generation have achieved remarkable visual realism but still struggle to obey basic physical laws such as gravity, inertia, and collision. Generated objects often move inconsistently across frames, exhibit implausible dynamics, or violate physical constraints, limiting the realism and reliability of AI-generated videos. We address this gap by introducing Physical Simulator In-the-loop Video Generation (PSIVG), a novel framework that integrates a physical simulator into the video diffusion process. Starting from a template video generated by a pre-trained diffusion model, PSIVG reconstructs the 4D scene and foreground object meshes, initializes them within a physical simulator, and generates physically consistent trajectories. These simulated trajectories are then used to guide the video generator toward spatio-temporally physically coherent motion. To further improve texture consistency during object movement, we propose a Test-Time Texture Consistency Optimization (TTCO) technique that adapts text and feature embeddings based on pixel correspondences from the simulator. Comprehensive experiments demonstrate that PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity. Project Page: https://vcai.mpi-inf.mpg.de/projects/PSIVG/
