Search papers, labs, and topics across Lattice.
This paper introduces Runtime Skill Audit (RSA), a dynamic analysis method designed to evaluate the security of agent skills in LLMs by examining their behavior under targeted runtime conditions. The method significantly improves upon static vetting approaches by profiling risk-relevant interfaces and preparing execution contexts, achieving a 90.0% accuracy with a notable reduction in false positives. RSA demonstrates resilience against self-evolving attacks, maintaining high detection rates of malicious skills even after multiple rounds of testing, highlighting its effectiveness in ensuring agent skill security.
Static vetting fails under evolving threats, but RSA maintains a 90% accuracy rate in detecting malicious agent skills through targeted runtime analysis.
Agent skills let LLM agents reuse instructions, resources, tools, and workflows, but they also create a new place for malicious behavior to hide. A skill may look benign in its documentation or code while becoming harmful only when it is invoked with particular user requests, local assets, persistent state, or multi-step tool interactions. This makes purely static vetting brittle. We present Runtime Skill Audit (RSA), a dynamic analysis method that audits skills by asking what the skill-mediated agent actually does under targeted runtime conditions. Instead of testing every skill with the same generic tasks, RSA profiles risk-relevant interfaces, prepares the execution context needed to exercise them, and assigns security labels from the resulting trace evidence. We instantiate RSA on OpenClaw and evaluate it on 100 skills against representative static baselines. RSA achieves 90.0\% accuracy with an 88.0\% true positive rate and an 8.0\% false positive rate, improving accuracy by 13.0 percentage points over the best static baseline. Under self-evolving attacks, static detectors collapse after one or two rounds, while RSA continues to detect 19--20 out of 20 malicious skills across rounds.
