Search papers, labs, and topics across Lattice.
This paper introduces V-LITE, a novel framework that leverages video generation models to estimate dynamic environment maps for photorealistic rendering by reframing lighting estimation as a guided video inpainting task. By incorporating a synthetic chrome ball into the scene, the model is guided to generate physically plausible reflections, effectively utilizing its inherent understanding of lighting. Extensive experiments show that V-LITE produces temporally coherent HDR environment maps, highlighting the dual capability of video diffusion models as both synthesizers and estimators of physical scene lighting.
Modern video generation models can inherently estimate dynamic lighting, producing HDR environment maps that are both temporally coherent and physically plausible.
Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.
