Search papers, labs, and topics across Lattice.
The paper introduces HarmoVid, a video diffusion model for relighting a foreground video to match a target background scene. To address the lack of paired video data for training, they generate a dataset by applying image-based harmonization models to videos and then introduce a lighting deflickering model to stabilize temporal inconsistencies. The model is trained on this deflickered data, along with an asymmetric alpha mask conditioning technique, to achieve high-quality video harmonization with strong temporal coherence and clean boundaries.
Say goodbye to jarring video composites: HarmoVid delivers temporally stable and physically plausible relighting for video portraits, outperforming frame-by-frame image harmonization.
We present a method for harmonizing the lighting of a foreground video to match a target background scene, adjusting shadows, color tone, and illumination intensity (relightful harmonization). Unlike images, acquiring labeled data for videos, where identical motions are recorded under different lighting conditions, is practically infeasible and non-scalable. While one way to create such paired data is to apply existing image-based harmonization models frame by frame to a video, the resulting outputs often suffer from significant temporal jitters. We overcome this problem by introducing a novel lighting deflickering model that can stabilize the global and local lighting flickering artifacts. Our video diffusion model learns from these upgraded deflickered data with a volume of real and synthetic videos to generate high-quality video harmonization results. We further propose an asymmetric alpha mask conditioning technique to learn the clean boundaries from real videos. Experiments demonstrate that our model achieves strong temporal coherence, naturalness, cleaner boundaries, and physically meaningful lighting behavior, while maintaining strong relighting expressiveness compared to prior image-based and video-based harmonization methods.
