Search papers, labs, and topics across Lattice.
PhysVideo, a two-stage video generation framework, addresses the challenge of physically plausible motion by first generating physics-aware orthogonal foreground videos using a novel physics-aware and geometry-enhanced cross-view attention mechanism, and then synthesizing full videos with background. This approach leverages a new dataset, PhysMV, comprising 40K scenes with four orthogonal viewpoints. Experiments demonstrate that PhysVideo significantly improves physical realism and spatio-temporal coherence compared to existing video generation methods.
Achieve more physically realistic video generation by explicitly modeling 3D geometry and physical attributes across multiple viewpoints.
Recent progress in video generation has led to substantial improvements in visual fidelity, yet ensuring physically consistent motion remains a fundamental challenge. Intuitively, this limitation can be attributed to the fact that real-world object motion unfolds in three-dimensional space, while video observations provide only partial, view-dependent projections of such dynamics. To address these issues, we propose PhysVideo, a two-stage framework that first generates physics-aware orthogonal foreground videos and then synthesizes full videos with background. In the first stage, Phys4View leverages physics-aware attention to capture the influence of physical attributes on motion dynamics, and enhances spatio-temporal consistency by incorporating geometry-enhanced cross-view attention and temporal attention. In the second stage, VideoSyn uses the generated foreground videos as guidance and learns the interactions between foreground dynamics and background context for controllable video synthesis. To support training, we construct PhysMV, a dataset containing 40K scenes, each consisting of four orthogonal viewpoints, resulting in a total of 160K video sequences. Extensive experiments demonstrate that PhysVideo significantly improves physical realism and spatial-temporal coherence over existing video generation methods. Home page: https://anonymous.4open.science/w/Phys4D/.
