Search papers, labs, and topics across Lattice.
This paper introduces TARS, a novel 3D-free video re-shooting paradigm that leverages text-driven semantic viewpoint specification to enhance control over camera motion and viewpoint. By conducting a timestep-wise sensitivity analysis, the authors identify that camera motion is predominantly established during high-noise stages, leading to the development of a self-supervised training approach that learns camera dynamics without the need for paired re-shooting data or 3D reconstruction. Extensive experiments demonstrate that TARS outperforms existing methods in terms of accuracy and temporal consistency in camera control, allowing for effective synthesis of unseen regions and perspective switching.
TARS achieves robust camera and viewpoint control in video re-shooting without relying on 3D reconstruction or paired data, enabling the plausible synthesis of previously unseen regions.
Video re-shooting aims to regenerate videos with controllable camera motion and viewpoint. Existing methods rely on explicit 3D priors, which are limited by reconstruction quality and often perform poorly when synthesizing previously unseen regions, or on paired videos with different camera trajectories, whose scarcity hinders generalization. We revisit video re-shooting through text-driven semantic viewpoint specification, enabling control over shot scale, viewing angle, and first-/third-person perspective. To this end, we propose TARS, a 3D-free video re-shooting paradigm. Timestep-wise sensitivity analysis reveals that camera motion is primarily established during high-noise stages, where coarse spatiotemporal structures are formed. Based on this insight, we introduce self-supervised training to learn camera dynamics and fundamental visual representations without paired re-shooting data or 3D reconstruction. Through data scaling and joint textual-camera conditioning, TARS supports robust camera and viewpoint control, plausibly synthesizing regions beyond the source view under large camera motions while enabling reverse-angle re-shooting and perspective switching. Extensive experiments show that TARS provides more accurate and temporally consistent camera control than prior methods. Project Page: https://ymlinfeng.github.io/TARS.github.io/
Tsinghua AI