Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the role of memory in end-to-end autonomous driving models by replacing real-time camera input with memories from prior drives at the same location. The findings reveal that using memory alone can achieve comparable or even superior performance on the NAVSIM benchmark, challenging the assumption that real-time scene perception is essential for high scores. However, the effectiveness of memory varies significantly across different benchmarks, highlighting the need for careful interpretation of performance metrics in autonomous driving evaluations.
Memory alone can rival real-time perception in autonomous driving, raising questions about the validity of current benchmark scores.
End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .