Search papers, labs, and topics across Lattice.
OpenSkillEval is introduced as an automated framework for evaluating both LLM agents augmented with skills and the skills themselves, addressing the need for dynamic, task-grounded evaluation in the rapidly expanding open-source skill ecosystem. The framework dynamically generates realistic task instances across five downstream applications and compares community-contributed skills under unified task settings. Experiments with 600 task instances and 30 open-source skills reveal that skill availability doesn't guarantee effective usage and that the benefit of skill augmentation is highly dependent on the model and agent framework used.
Turns out, many publicly available LLM skills don't consistently outperform base agents, highlighting the critical need for rigorous, task-grounded evaluation in the open-source skill ecosystem.
Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improving agent performance on real-world downstream tasks. However, as the open-source skill ecosystem rapidly expands, it remains unclear how different models and agent frameworks interact with skills, how to evaluate skill quality, and how users should select skills under practical cost-performance trade-offs. In this paper, we present \textsc{OpenSkillEval}, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves. Instead of relying on static benchmarks, \textsc{OpenSkillEval} automatically constructs realistic task instances from evolving real-world artifacts across five categories of downstream applications: presentation generation, front-end web design, poster generation, data visualization, and report generation. It further collects and organizes community-contributed skills for controlled comparison under unified task settings. Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks. Our results show that skill availability does not guarantee effective skill usage, that the benefit of skill augmentation depends strongly on both the underlying model and the agent framework, and that many publicly popular skills do not consistently outperform base agents without skills. These findings highlight the need for dynamic, task-grounded evaluation and provide practical insights into the design, selection, and deployment of skills for LLM agents. Additional cases and benchmark resources are available on the project website: https://yingjiahao14.github.io/OpenSkillEval-Web/.
