Search papers, labs, and topics across Lattice.
The authors introduce UENR-600K, a large-scale (600K frame pairs) synthetic dataset for nighttime video deraining, generated using Unreal Engine to simulate physically realistic 3D rain particles and their interactions with artificial lighting. This addresses the limitations of existing datasets that rely on 2D rain overlays and fail to capture the complex optical phenomena of nighttime rain. By training a video-to-video generation model (Wan 2.2) on UENR-600K, the authors achieve state-of-the-art performance and demonstrate significantly improved generalization to real-world nighttime video deraining scenarios.
Nighttime video deraining models can now generalize to the real world, thanks to a massive, physically-grounded synthetic dataset that finally captures the complex interplay of rain and artificial light.
Nighttime video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2 video generation model. Our baseline treat deraining as a video-to-video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our dataset generalize significantly better to real-world videos. Project page: https://showlab.github.io/UENR-600K/.
