Search papers, labs, and topics across Lattice.
This paper introduces MaliciousSkillBench, a comprehensive benchmark designed to enhance the detection of malicious agent skills by consolidating 13 public sources into a unified dataset of 9,740 skills, including 7,505 malicious and 2,235 benign entries. The authors identify significant discrepancies in threat composition across sources, which complicates detection efforts, and evaluate various detection methods, revealing that while learned detectors achieve high F1 scores, they struggle with source-disjoint evaluations, leading to a high false positive rate. The findings underscore the necessity for improved benchmark coverage and evaluation methodologies to effectively balance malicious detection and benign over-flagging.
Reliable detection of malicious agent skills hinges on a comprehensive benchmark that reveals stark differences in threat composition across sources.
Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a direct distribution channel for malicious behavior, yet existing malicious-Skill datasets are fragmented across sources, artifact formats, evidence regimes, and benign coverage; duplicated and structurally related content further complicates direct aggregation and evaluation. We present MaliciousSkillBench, a comprehensive benchmark for malicious Agent Skill detection. We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families. After conservative cross-label conflict exclusion, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign. To characterize its coverage, we harmonize 11 attack categories for 4,983 malicious identities with supported source-native mappings and find substantial differences in threat composition across sources. We then evaluate three learned text detectors and three off-the-shelf Skill scanners. Learned detectors achieve 0.882-0.932 Random Macro-F1 but only 0.653-0.665 under Source-Disjoint evaluation; the strongest word TF-IDF SVM scores 0.932/0.916/0.665 on Random/structural-disjoint/Source-Disjoint while retaining 95.6% malicious recall but producing 62.4% benign FPR on held-out sources. Off-the-shelf scanners occupy different but also unsatisfactory operating regimes, reducing false positives only at the cost of sharply lower malicious recall. Together, these results show that reliable malicious-Skill detection requires both broader cross-source benchmark coverage and evaluation that jointly measures attack detection and benign over-flagging.
