Search papers, labs, and topics across Lattice.
This paper introduces a framework for evaluating the spatio-temporal reasoning capabilities of frozen LLMs in vehicle trajectory prediction by encoding traffic agent trajectories and HD maps into LLM-compatible tokens. A traffic encoder and CNN extract spatial features from trajectories and maps, respectively, which are then transformed into tokens via a reprogramming adapter for the LLM. Experiments demonstrate the framework's ability to quantitatively analyze the impact of multi-modal information, particularly map semantics, on trajectory prediction accuracy across various LLM architectures.
Frozen LLMs, when fused with spatial scene encodings, can effectively reason about vehicle trajectories, opening new avenues for integrating language-based reasoning into autonomous driving systems.
Large language models (LLMs) have recently demonstrated strong reasoning capabilities and attracted increasing research attention in the field of autonomous driving (AD). However, safe application of LLMs on AD perception and prediction still requires a thorough understanding of both the dynamic traffic agents and the static road infrastructure. To this end, this study introduces a framework to evaluate the capability of LLMs in understanding the behaviors of dynamic traffic agents and the topology of road networks. The framework leverages frozen LLMs as the reasoning engine, employing a traffic encoder to extract spatial-level scene features from observed trajectories of agents, while a lightweight Convolutional Neural Network (CNN) encodes the local high-definition (HD) maps. To assess the intrinsic reasoning ability of LLMs, the extracted scene features are then transformed into LLM-compatible tokens via a reprogramming adapter. By residing the prediction burden with the LLMs, a simpler linear decoder is applied to output future trajectories. The framework enables a quantitative analysis of the influence of multi-modal information, especially the impact of map semantics on trajectory prediction accuracy, and allows seamless integration of frozen LLMs with minimal adaptation, thereby demonstrating strong generalizability across diverse LLM architectures and providing a unified platform for model evaluation.
