Search papers, labs, and topics across Lattice.
This paper introduces DrivingAgent, a novel framework specifically designed to tackle the dual challenges of system design and real-time scheduling in autonomous driving. By automating module development through system architecture interpretation and super-network training, and employing a lightweight LLM for dynamic scheduling, DrivingAgent effectively addresses the limitations of existing frameworks. Experimental results show that it significantly improves the speed-accuracy trade-off on the nuScenes and Bench2Drive benchmarks, highlighting its potential for enhancing autonomous driving systems.
DrivingAgent achieves a superior speed-accuracy trade-off in autonomous driving, revolutionizing how we design and schedule driving systems.
Many autonomous driving systems are increasingly incorporating foundation models to improve generalization and handle long-tail scenarios. However, this trend introduces two key challenges: (i) the manual and labor-intensive process of designing and integrating new models, and (ii) the lack of intelligent, dynamic scheduling mechanisms to meet strict real-time constraints. While Large Language Model (LLM)-based agents offer a promising avenue for automation, existing frameworks are ill-suited for autonomous driving. Specifically, they fail to distinguish between the fundamentally different requirements of system design and real-time scheduling, treat modules as opaque black boxes, and are not designed for continuous operation. To address these limitations, we propose DrivingAgent, a novel agent framework tailored to the dual challenges of autonomous driving system design and scheduling. In the design phase, DrivingAgent automates module development by interpreting system architecture, generating code, and validating modules via super-network training. In the scheduling phase, it employs a lightweight LLM trained with reinforcement learning to dynamically orchestrate system modules in real time, supported by a structured memory that integrates long-term storage with timestamped short-term context. Experimental results demonstrate that DrivingAgent achieves a superior speed--accuracy trade-off on both the nuScenes and Bench2Drive benchmarks.
