Search papers, labs, and topics across Lattice.
This study evaluates the performance of seven open-source large language models (LLMs) in classifying complex Cyber Threat Intelligence (CTI) reports according to the MITRE ATT&CK framework, using a newly constructed dataset of 2,076 human-annotated sentences. The research reveals that the highest-performing LLM achieved a micro-averaged F1 score of only 0.22, highlighting the inadequacy of current models for effective multi-label ATT&CK classification in real-world scenarios. Additionally, the findings indicate that while larger models correlate with better performance, prompt strategy and temperature settings do not yield significant improvements, establishing a critical baseline for future research in this domain.
Current open-source LLMs fall short in effectively classifying complex CTI reports, with the best model only achieving an F1 score of 0.22.
Classifying Cyber Threat Intelligence (CTI) using MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is essential for proactive defense, but historically required extensive human effort. Pre-Large Language Model (LLM) automation sped up this process, but could not resolve the complex language and multi-step attack patterns found in unstructured CTI reports. LLMs addressed previous limitations by using contextual reasoning to understand unstructured text. However, current evaluations rely on simplified, single-technique sentences that ignore the complexity of real-world CTI reports, which often leads to inflated performance results. Consequently, the baseline performance of open-source LLMs on complex unstructured CTI reports remains unevaluated. To address this gap, we constructed a ground-truth dataset of 2,076 human-annotated sentences (1,281 technique-positive, 795 negative) from 83 complex unstructured CTI reports. These sentences were mapped to 114 unique ATT&CK techniques using a six-phase annotation process, achieving \k{appa} = 0.68 inter-annotator agreement. Using this dataset, we evaluated seven open-source LLMs ranging from 8B to 236B parameters across prompt strategy and temperature configurations. The highest-performing LLM achieved a micro-averaged F1 score of 0.22, establishing the empirical baseline for multi-label ATT&CK classification on complex unstructured CTI. Parameter size showed a statistically significant positive correlation with F1 score. Prompt strategy and temperature produced no statistically significant gains across model configurations. These results indicate that current open-source LLMs are insufficient for production-grade ATT&CK classification. The dataset, benchmark, and findings provide a reproducible foundation for future CTI research.
