Search papers, labs, and topics across Lattice.
The paper introduces CAN-QA, a novel question-answering benchmark for evaluating reasoning capabilities of large language models (LLMs) on in-vehicle Controller Area Network (CAN) traffic analysis. The benchmark converts raw CAN logs into temporally segmented windows and generates natural-language questions with rule-based templates, creating 33,128 QA pairs across 10 categories. Experiments using CAN-QA reveal that while LLMs capture superficial patterns in CAN traffic, they struggle with temporal reasoning, multi-condition inference, and high-level behavioral interpretation, highlighting limitations in their ability to perform forensic analysis of CAN bus data.
LLMs can parrot CAN bus data, but CAN-QA reveals they fail at the temporal reasoning and multi-condition inference needed for real-world vehicle security forensics.
The Controller Area Network (CAN) is a safety-critical in-vehicle communication protocol that lacks built-in security mechanisms, making intrusion detection essential. Existing approaches predominantly formulate CAN intrusion detection as a classification task, mapping complex traffic patterns to attack labels. However, this formulation abstracts away the temporal and relational structure of CAN traffic and misaligns with real-world forensic workflows, which require systematic reasoning about traffic behavior. To address this gap, we introduce CAN-QA, the first benchmark that reformulates CAN traffic analysis as a question-answering (QA) task. CAN-QA converts raw CAN logs into temporally segmented windows and applies deterministic rule-based templates to generate natural-language questions paired with automatically derived ground-truth answers. The resulting dataset comprises 33,128 QA pairs across 10 categories, each targeting distinct semantic and temporal properties of CAN traffic. Using CAN-QA, we evaluate large language models across both True/False and multiple-choice formats. Our results indicate that, although these models capture superficial statistical regularities, they struggle with temporal reasoning, multi-condition inference, and higher-level behavioral interpretation. Our code is available at https://github.com/Kriiiiss/CAN-QA.
