Search papers, labs, and topics across Lattice.
GSDrive improves end-to-end autonomous driving policies by using 3D Gaussian Splatting (3DGS) to create a differentiable environment for physics-based reward shaping. They incorporate a flow matching-based trajectory predictor within the 3DGS simulator for multi-modal trajectory probing, enabling the assessment of prospective rewards. This provides dense, immediate feedback to the RL agent, leading to improved performance compared to sparse, event-based reward signals.
Forget sparse rewards, GSDrive unlocks dense, physics-based reward shaping for autonomous driving by rendering the environment with differentiable 3D Gaussian Splatting.
End-to-end (E2E) autonomous driving presents a promising approach for translating perceptual inputs directly into driving actions. However, prohibitive annotation costs and temporal data quality degradation hinder long-term real-world deployment. While combining imitation learning (IL) and reinforcement learning (RL) is a common strategy for policy improvement, conventional RL training relies on delayed, event-based rewards-policies learn only from catastrophic outcomes such as collisions, leading to premature convergence to suboptimal behaviors. To address these limitations, we introduce GSDrive, a framework that exploits 3D Gaussian Splatting (3DGS) for differentiable, physics-based reward shaping in E2E driving policy improvement. Our method incorporates a flow matching-based trajectory predictor within the 3DGS simulator, enabling multi-mode trajectory probing where candidate trajectories are rolled out to assess prospective rewards. This establishes a bidirectional knowledge exchange between IL and RL by grounding reward functions in physically simulated interaction signals, offering immediate dense feedback instead of sparse catastrophic events. Evaluated on the reconstructed nuScenes dataset, our method surpasses existing simulation-based RL driving approaches in closed-loop experiments. Code is available at https://github.com/ZionGo6/GSDrive.
