Search papers, labs, and topics across Lattice.
This paper introduces a comprehensive evaluation framework for assessing agent skills in large language models (LLMs), addressing the lack of standardized methodologies in the field. By constructing realistic tasks and applying the framework to 500 real-world skills, the authors reveal significant variability in model performance based on skill integration, highlighting the impact of structured knowledge artifacts on agent behavior. The findings underscore the necessity of skill evaluation in optimizing LLM capabilities, with the release of a dataset to facilitate further research in this area.
Models exhibit drastic performance variations when utilizing agent skills, with some configurations gaining substantial advantages while others falter.
Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.
