Search papers, labs, and topics across Lattice.
This paper introduces a framework for autonomous vehicles that translates open-ended natural language instructions into executable control signals by using an LLM to generate scripts that schedule multiple MPC-based motion planners. The scheduling-centric design decouples semantic reasoning from vehicle control, creating a transparent and traceable decision-making chain. Experiments demonstrate improved task completion rates, reduced LLM query costs, and comparable safety to specialized AD approaches, while also exhibiting tolerance to LLM inference latency.
Forget brittle, end-to-end autonomous driving systems: this LLM-orchestrated multi-planner architecture achieves better task completion and safety with improved interpretability.
Most Human-Machine Interaction (HMI) research overlooks the maneuvering needs of passengers in autonomous driving (AD). Natural language offers an intuitive interface, yet translating passenger open-ended instructions into control signals, without sacrificing interpretability and traceability, remains a challenge. This study proposes an instruction-realization framework that leverages a large language model (LLM) to interpret instructions, generates executable scripts that schedule multiple model predictive control (MPC)-based motion planners based on real-time feedback, and converts planned trajectories into control signals. This scheduling-centric design decouples semantic reasoning from vehicle control at different timescales, establishing a transparent, traceable decision-making chain from high-level instructions to low-level actions. Due to the absence of high-fidelity evaluation tools, this study introduces a benchmark for open-ended instruction realization in a closed-loop setting. Comprehensive experiments reveal that the framework significantly improves task-completion rates over instruction-realization baselines, reduces LLM query costs, achieves safety and compliance on par with specialized AD approaches, and exhibits considerable tolerance to LLM inference latency. For more qualitative illustrations and a clearer understanding.
