Search papers, labs, and topics across Lattice.
This paper conducts a reproducibility audit of knowledge graph extraction systems used for threat reports, revealing significant discrepancies in reported triple-F1 scores due to varying matching protocols. By re-evaluating ten systems under eight different scoring methods, the authors found that traditional matchers consistently underperformed compared to a large language model (LLM) judge, which achieved an 86% agreement with multi-reviewer adjudication. The study introduces CTIForge, a validation tool that enhances precision across various deployment configurations while highlighting the complex interplay between extraction methods and validation processes.
Traditional knowledge graph matchers fall short, with LLMs outperforming them by a significant margin in threat report evaluations.
Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16-0.70 F1. On GRID's external 378-item calibration set, no mechanical matcher (lexical, embedding, or entailment) agrees with multi-reviewer adjudication above 71%, whereas an LLM judge reaches 86%. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, validation raises precision for all four hosted backbones and lowers it for all three offline backbones. Because backbone, decoding, and backend-specific prompting covary, this is a descriptive split rather than an isolated serving effect. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.
