Search papers, labs, and topics across Lattice.
This paper introduces PhysRAG, a novel pipeline that enhances the physical awareness of video generation models through a Retrieval-Augmented Generation (RAG) approach. By implementing a two-stage data filtering pipeline on the WISA-80K dataset, the authors curate a high-quality training set of 7K videos and develop a physical video database to inject physical knowledge into a video diffusion model. The resulting method achieves state-of-the-art performance in visual quality and adherence to physical rules, outperforming existing benchmarks like PhyGenBench and VBench.
PhysRAG not only elevates visual fidelity in video generation but also ensures compliance with physical laws, setting a new benchmark for realism in AI-generated content.
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K dataset, resulting in a curated set of 7K high-quality videos for training. Furthermore, we construct a physical video database and develop a mechanism to inject physical knowledge into a video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.
