Search papers, labs, and topics across Lattice.
To assess the viability of language models in cyber threat intelligence, the authors benchmarked eight LLMs on operational threat level determination using a curated dataset derived from public MISP OSINT feeds under both zero-shot and supervised fine-tuning (SFT) regimes. Zero-shot evaluations exposed severe failure modes in assessing threat severity, whereas SFT drove noticeable gains, yielding F1 scores between 0.40 and 0.58 across architectures. These findings indicate that while domain adaptation is essential, current model capabilities remain well below the reliability threshold required for autonomous SecOps deployment.
Even after targeted fine-tuning, LLMs peak at an F1 score of just 0.58 on cyber threat level determination, revealing that current models remain far too brittle for operational SecOps workflows.
The fast progress of large language models (LLMs) opens new opportunities in the management of cyber threat intelligence, but their reliability for operational tasks remains unclear. In this work, we benchmark LLMs on the task of threat level determination. First, we construct a curated dataset derived from publicly available MISP OSINT feeds. Next, we design a tailored prompt to systematically compare eight different LLMs under zero-shot conditions. Finally, we apply supervised fine-tuning on each model and perform a comparative analysis between baseline and fine-tuned versions. Our results show that zero-shot models achieve weak performance, with limited ability to correctly assign threat levels. Fine-tuned models, however, demonstrate substantial improvements, reaching F1 scores between 0.40 and 0.58 depending on the base architecture. Despite this progress, the performance is still low for practical deployment, highlighting the need for additional research on data quality, model adaptation, and domain-specific tuning.
