Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors audited eight cybersecurity benchmarks across ten proprietary, open-weight, and specialized LLMs by formalizing benchmark evaluation as a configurable measurement pipeline. They identified 15 systematic pipeline failure modes and discovered that isolated pipeline choices can shift an individual model's score by over 80 percentage points, causing nine out of ten models to shift by at least three leaderboard ranks under standardized settings. These results reveal that existing cybersecurity benchmark scores frequently reflect evaluation harness conventions rather than genuine model capabilities, undermining current empirical assessments in AI security.
A single pipeline configuration choice can swing a model's benchmark score by more than 80 percentage points, revealing that most cybersecurity LLM rankings reflect harness implementation quirks rather than actual capability.
Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.