Search papers, labs, and topics across Lattice.
Cam2Sim is a novel tool that bridges the gap between real-world driving data and synthetic simulation environments by reconstructing detailed driving scenarios from camera images and poses. By employing Gaussian Splatting for rendering, it enhances the visual fidelity of simulated outputs to closely match real-world observations, while also enabling the construction of comprehensive driving scenarios in the CARLA simulator. Validation against both a simulation-only baseline and real-world data demonstrates that Cam2Sim significantly improves the quality of scenario reconstruction and the behavioral fidelity of autonomous driving models in closed-loop testing.
Gaussian Splatting transforms real-world driving footage into high-fidelity simulation scenarios, drastically improving the realism of autonomous driving tests.
Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI
