Search papers, labs, and topics across Lattice.
This paper introduces SKILLVETBENCH, a novel vetting system that leverages an LLM-as-Judge to evaluate the security risks of community-contributed skills in open-source LLM agent ecosystems. By employing a five-dimensional Skill Agentic Risk Score (SARS) and integrating it with CVSS v4.0 metrics, the system effectively identifies multi-agent and instruction-layer risks that traditional code-based scanners overlook. The results demonstrate that SKILLVETBENCH achieves zero false negatives and false positives in its evaluations, significantly outperforming existing static baselines in detecting malicious skills.
An LLM-as-Judge can achieve zero false negatives in identifying malicious skills, revealing critical vulnerabilities that traditional scanners miss.
Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted. The gap we fill: existing scanners operate at the code layer and are structurally blind to instruction-layer and multi-agent risk - natural-language directives that hijack an agent, exfiltrate data through encoded side channels, or chain harm across pipelines - so what is needed is a semantic, multi-dimensional vetting system rather than another signature matcher. We present SKILLVETBENCH, a live public leaderboard on Hugging Face that uses an LLM-as-Judge to vet agent skills. What is new: SARS (Skill Agentic Risk Score), a five-dimensional agentic-risk metric with a principled weighted formula for instruction-following systems. What is integrated: full CVSS v4.0 vector decomposition and a ClawHub dual-view that places our LLM-generated review beside the official marketplace verdict. What is demonstrated: drawing on our companion benchmark paper [ 1], the LLM-as-Judge stage achieves zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls, while the best static baseline (SKILLSIEVE) still misses 15%; for instruction-layer categories such as Prompt Injection and Memory Poisoning, conventional tools miss between 89% and 100% of threats (e.g., CODEBERT detects none of nine memory-poisoning skills). Detection rates vary from 35% to 95% across four LLM evaluators, motivating ensemble scoring in production deployments.
