Search papers, labs, and topics across Lattice.
This paper introduces SkillsMetric, a comprehensive five-stage static analysis framework designed to evaluate the security of agent skills used in LLM-based systems. By analyzing a dataset of 2,266 skills across various attack types, the framework achieved an impressive AUC of 0.93 and a 5-fold cross-validated F1 score of 73.4%, effectively detecting threats like data exfiltration and steganographic payloads. However, it also reveals critical vulnerabilities, as certain attack vectors, such as host destruction and prompt injection, remain undetected, highlighting the limitations of static analysis in ensuring skill security.
Static analysis can miss critical attack vectors, with host destruction attacks evading detection entirely and prompt injection only partially identified.
Agent Skills---structured packages of instructions and scripts that augment LLM-based agents---are rapidly proliferating, yet their security properties remain under-explored. We present \textsc{SkillsMetric}, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions. We construct an adversarial evaluation dataset of 2{,}266 skills spanning 16~attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus. Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4\%$\pm$0.5\%, with strong detection of data exfiltration (93\%) and steganographic payloads (93\%). Crucially, we identify fundamental blind spots: \emph{host destruction} attacks using common shell commands evade all five stages (0\% detection), and \emph{prompt injection} via natural-language manipulation achieves only 42\% detection. These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review.
