Search papers, labs, and topics across Lattice.
This paper introduces SkillSentry, a dynamic safety-testing framework that utilizes adaptive honey worlds to evaluate the conditional behaviors of external skills in large language model agents. By simulating environments with controlled decoy resources and adaptively generating tasks, SkillSentry effectively identifies harmful behaviors that static analysis methods often miss, achieving a remarkable 99.50% Recall and 96.26% average F1 score on standard benchmarks. In scenarios involving semantics-preserving evasion, SkillSentry outperforms existing scanners, demonstrating its robustness in detecting potential threats in agent skills.
SkillSentry reveals that dynamic testing can uncover harmful agent behaviors that static analysis fails to detect, achieving unprecedented accuracy in safety evaluations.
External skills extend the capabilities of large language model agents, but also introduce an execution-time attack surface: a skill that appears benign under inspection may reveal harmful behavior only after particular environmental states, resources, or interaction histories are encountered. Existing scanners primarily rely on static analysis, predefined rules, or one-shot semantic judgments, making such conditional behavior difficult to elicit and attribute. We present SkillSentry, a dynamic safety-testing framework based on adaptive honey worlds. SkillSentry infers the intended capability boundary of a skill, constructs an LLM-simulated environment with controlled decoy resources, and adaptively generates tasks to explore its behavioral states. It then compares skill-enabled trajectories with matched no-skill executions, grounding suspicious behaviors in source code and verified execution traces before making a final decision. We evaluate SkillSentry against seven scanner configurations. SkillSentry achieves 99.50% Recall and 96.26% average F1 on standard benchmarks. Under semantics-preserving evasion, it reaches 92.95% average F1, compared with 80.07% for the strongest baselines. Our code is available at https://github.com/nizhangli062-jpg/SkillSentry-Adaptive-Honey-Worlds-for-Dynamic-Safety-Testing-of-Agent-Skills.
Tsinghua AI