Search papers, labs, and topics across Lattice.
The paper introduces KITScenes LongTail, a new dataset focused on rare driving scenarios, featuring multi-view video, trajectories, instructions, and expert-annotated reasoning traces in multiple languages. This dataset enables the training and evaluation of multimodal models on instruction following and semantic coherence in challenging, long-tail driving situations. The multilingual reasoning traces from diverse experts provide a unique resource for studying the impact of different reasoning styles on driving competence.
Training data is not enough: reasoning traces from diverse cultural backgrounds are critical for safe and reliable autonomous driving in rare, long-tail scenarios.
In real-world domains such as self-driving, generalization to rare scenarios remains a fundamental challenge. To address this, we introduce a new dataset designed for end-to-end driving that focuses on long-tail driving events. We provide multi-view video data, trajectories, high-level instructions, and detailed reasoning traces, facilitating in-context learning and few-shot generalization. The resulting benchmark for multimodal models, such as VLMs and VLAs, goes beyond safety and comfort metrics by evaluating instruction following and semantic coherence between model outputs. The multilingual reasoning traces in English, Spanish, and Chinese are from domain experts with diverse cultural backgrounds. Thus, our dataset is a unique resource for studying how different forms of reasoning affect driving competence. Our dataset is available at: https://hf.co/datasets/kit-mrt/kitscenes-longtail
