Search papers, labs, and topics across Lattice.
The paper introduces AgentDrive, a large-scale (300K scenarios) open benchmark dataset for training and evaluating LLM-based autonomous agents in driving scenarios. The dataset is generated using an LLM-driven prompt-to-JSON pipeline that formalizes a factorized scenario space across seven orthogonal axes and validates scenarios against physical and schema constraints. The authors also introduce AgentDrive-MCQ, a 100K-question multiple-choice benchmark, and evaluate 50 LLMs, finding that proprietary models excel in contextual and policy reasoning while open models are catching up in structured and physics-grounded reasoning.
LLMs in autonomous driving get a reality check: AgentDrive, a new benchmark, exposes gaps in reasoning abilities, particularly in physics and structured scenarios, even as open models rapidly improve.
The rapid advancement of large language models (LLMs) has sparked growing interest in their integration into autonomous systems for reasoning-driven perception, planning, and decision-making. However, evaluating and training such agentic AI models remains challenging due to the lack of large-scale, structured, and safety-critical benchmarks. This paper introduces AgentDrive, an open benchmark dataset containing 300,000 LLM-generated driving scenarios designed for training, fine-tuning, and evaluating autonomous agents under diverse conditions. AgentDrive formalizes a factorized scenario space across seven orthogonal axes: scenario type, driver behavior, environment, road layout, objective, difficulty, and traffic density. An LLM-driven prompt-to-JSON pipeline generates semantically rich, simulation-ready specifications that are validated against physical and schema constraints. Each scenario undergoes simulation rollouts, surrogate safety metric computation, and rule-based outcome labeling. To complement simulation-based evaluation, we introduce AgentDrive-MCQ, a 100,000-question multiple-choice benchmark spanning five reasoning dimensions: physics, policy, hybrid, scenario, and comparative reasoning. We conduct a large-scale evaluation of fifty leading LLMs on AgentDrive-MCQ. Results show that while proprietary frontier models perform best in contextual and policy reasoning, advanced open models are rapidly closing the gap in structured and physics-grounded reasoning. We release the AgentDrive dataset, AgentDrive-MCQ benchmark, evaluation code, and related materials at https://github.com/maferrag/AgentDrive
