Search papers, labs, and topics across Lattice.
This paper explores methods for reducing the sim2real appearance gap in synthetic datasets generated by game engines, which limits their applicability to real-world computer vision tasks. They compare a diffusion model (FLUX.2-4B Klein) against an image-to-image translation model (REGEN) for improving photorealism. The authors then propose and validate a hybrid approach combining both models, demonstrating that it achieves superior visual realism while preserving semantic consistency compared to using either model alone.
Combining diffusion models with image-to-image translation yields surprisingly realistic synthetic data, outperforming either method alone in closing the sim2real gap.
Video game engines have been an important source for generating large volumes of visual synthetic datasets for training and evaluating computer vision algorithms that are to be deployed in the real world. While the visual fidelity of modern game engines has been significantly improved with technologies such as ray-tracing, a notable sim2real appearance gap between the synthetic and the real-world images still remains, which limits the utilization of synthetic datasets in real-world applications. In this letter, we investigate the ability of a state-of-the-art image generation and editing diffusion model (FLUX.2-4B Klein) to enhance the photorealism of synthetic datasets and compare its performance against a traditional image-to-image translation model (REGEN). Furthermore, we propose a hybrid approach that combines the strong geometry and material transformations of diffusion-based methods with the distribution-matching capabilities of image-to-image translation techniques. Through experiments, it is demonstrated that REGEN outperforms FLUX.2-4B Klein and that by combining both FLUX.2-4B Klein and REGEN models, better visual realism can be achieved compared to using each model individually, while maintaining semantic consistency. The code is available at: https://github.com/stefanos50/Hybrid-Sim2Real
