Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SkillSieve is introduced as a three-layered framework for detecting malicious skills in AI agent marketplaces, addressing the limitations of regex scanners and static analyzers. It uses XGBoost for initial filtering, followed by parallel LLM-based analysis of suspicious skills across intent, permissions, covert behavior, and consistency, and finally employs a jury of LLMs for high-risk cases. Evaluated on a large dataset of real-world skills, SkillSieve achieves a significantly higher F1 score (0.800) compared to existing methods (0.421) while maintaining low cost.
A hierarchical triage system slashes the cost of auditing AI agent skills by first filtering with XGBoost, then strategically deploying LLMs for targeted analysis, and finally using a jury of LLMs for debate.
OpenClaw's ClawHub marketplace hosts over 13,000 community-contributed agent skills, and between 13% and 26% of them contain security vulnerabilities according to recent audits. Regex scanners miss obfuscated payloads; formal static analyzers cannot read the natural language instructions in SKILL.md files where prompt injection and social engineering attacks hide. Neither approach handles both modalities. SkillSieve is a three-layer detection framework that applies progressively deeper analysis only where needed. Layer 1 runs regex, AST, and metadata checks through an XGBoost-based feature scorer, filtering roughly 86% of benign skills in under 40ms on average at zero API cost. Layer 2 sends suspicious skills to an LLM, but instead of asking one broad question, it splits the analysis into four parallel sub-tasks (intent alignment, permission justification, covert behavior detection, cross-file consistency), each with its own prompt and structured output. Layer 3 puts high-risk skills before a jury of three different LLMs that vote independently and, if they disagree, debate before reaching a verdict. We evaluate on 49,592 real ClawHub skills and adversarial samples across five evasion techniques, running the full pipeline on a 440 ARM single-board computer. On a 400-skill labeled benchmark, SkillSieve achieves 0.800 F1, outperforming ClawVet's 0.421, at an average cost of 0.006 per skill. Code, data, and benchmark are open-sourced.