Search papers, labs, and topics across Lattice.
This paper introduces DiagChain, a diagnostic benchmark designed to evaluate LLM agents on their ability to reconstruct evidence-grounded attack chains through a stage-wise assessment. By utilizing the MAIN-69 suite of scenarios, the authors identify critical weaknesses in LLM performance, revealing that even the best models only achieve a 39.6% success rate on reference steps. The findings highlight the necessity of detailed diagnostic metrics to understand and enhance LLM capabilities in cybersecurity contexts, particularly in the integration and ordering of retrieved evidence.
LLMs struggle to effectively integrate and order evidence in attack chain reconstruction, with top models only succeeding 39.6% of the time on critical tasks.
Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly evaluate final outputs or aggregate accuracy, providing limited insight into how errors arise and propagate across intermediate reasoning stages. We present DiagChain, a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. DiagChain includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It further introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics are introduced to assess distinct stages of the reconstruction process and support systematic failure diagnosis. Based on evaluations using 6 LLMs, DiagChain reveals that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Our analysis further shows that smaller models struggle with the more basic task of incorporating retrieved evidence into their outputs, whereas larger models can proceed to later steps, where correctly ordering that evidence becomes the main bottleneck. These results validate the importance of diagnostic evaluation beyond end-to-end accuracy and provide actionable insights for improving evidence-grounded cybersecurity agents.
Tsinghua AI