Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses alert fatigue in Security Operations Centers by training a chain-of-thought (CoT) reasoning-enabled triage classifier on real Windows endpoint detections, enhancing the model's ability to assess genuine threats. By integrating automated prompt optimization, self-training, and reinforcement learning with verifiable rewards, the system achieves an impressive 82.6% test accuracy. Notably, it improves benign recall by 43.0% and malicious recall by 18.3% compared to traditional direct-label LLM classifiers, highlighting the importance of a separate calibrator for effective confidence estimation.
CoT reasoning not only boosts detection accuracy but also reveals that a dedicated calibrator is essential for maintaining high-confidence recall in cybersecurity triage.
A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day. Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat. We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections by combining automated prompt optimization, self-training, and reinforcement learning with verifiable rewards. We find that CoT reasoning also degrades the label-token probabilities that automated triage relies on, so we separately train a calibrator that reads the full reasoning trace and estimates the probability that the verdict is correct. Our system reaches 82.6% test accuracy and, at the high-confidence operating point that governs automated triage, improves benign recall by 43.0% and malicious recall by 18.3% over a direct-label LLM classifier. We further show that the trained calibrator is necessary - an untrained confidence judge collapses high-confidence recall to zero - and that a finetuned 30B model significantly outperforms frontier general-purpose models, motivating targeted training over scale.