Search papers, labs, and topics across Lattice.
This paper introduces VPT, a fine-tuning framework designed to enhance physical consistency in video diffusion models by implementing role-aware joint training and modality-decoupled denoising. By categorizing entities into distinct roles and allowing independent noise levels for visual and auxiliary modalities, VPT effectively mitigates recursive prediction errors and improves the modeling of physical dynamics. Experimental results demonstrate significant improvements in physical consistency metrics, achieving relative gains of 39.4% in scene accuracy and 17.9% in physical consistency on the VideoPhy benchmark compared to existing models.
Role-aware training can boost video diffusion models' physical consistency by up to 39.4% without sacrificing visual fidelity.
While modern video diffusion models excel in visual fidelity, maintaining long-range physical consistency remains a formidable challenge. Conventional pixel-reconstruction objectives mainly focus on appearance details and often fail to capture the underlying dynamics of a scene. To mitigate this, recent efforts have integrated auxiliary modalities (e.g., optical flow) to introduce physics priors via joint training with video appearance. However, these methods have three main limitations: (1) they do not distinguish the different motion patterns of different entity types; (2) joint modeling of visual and auxiliary modalities can cause capacity conflicts and weaken the pretrained visual prior; and (3) auxiliary modalities may accumulate errors during inference. To address these issues, we propose \textbf{VPT}, a fine-tuning framework for improving physical consistency in video diffusion models. VPT introduces a role-aware signal that groups entities into agents, controlled objects, passive objects, and background, so that different physical roles can be modeled more clearly. We further propose a modality-decoupled denoising strategy, where the visual and auxiliary channels are assigned independent noise levels. Together with a loss-weight decay strategy, this design makes auxiliary modalities serve as soft constraints rather than strong dependencies, mitigating recursive prediction errors during inference. We also introduce cross-step auto-guidance to further strengthen physical dynamics. Experiments show that VPT improves physical consistency while preserving visual quality, achieving relative gains of 39.4\% in SA and 17.9\% in PC on VideoPhy benchmark over Wan2.1-T2V-1.3B, and consistent improvements on VideoPhy-2 benchmark. The project page is available at https://tom-zgt.github.io/VPT.
