Search papers, labs, and topics across Lattice.
The paper introduces LightCrafter, a novel hybrid pipeline for video relighting that combines physically-based rendering (PBR) with diffusion models to enhance temporal consistency and control over lighting effects. By translating a PBR rendering of the input video under target illumination instead of the raw video, the method effectively integrates illumination targets, improving the model's ability to handle complex lighting scenarios like global illumination. Experimental results demonstrate that LightCrafter surpasses existing state-of-the-art methods on real-world relighting benchmarks while also providing a new synthetic benchmark for further research.
LightCrafter achieves superior video relighting by integrating PBR with diffusion models, enabling intricate lighting control and long-form temporal consistency without the need for extensive training data.
Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.
CMU ML