Search papers, labs, and topics across Lattice.
This paper introduces Skill-Use, a benchmark designed to evaluate how effectively large language model (LLM) agents can recognize and apply skills in agentic harnesses. The study reveals that while LLMs can be evaluated on their ability to trigger relevant skills and comply with procedures, their performance remains inconsistent, with the best models achieving a Skill-Use score of only 0.613. The findings indicate that skill use is not a stable attribute of the models but rather depends on the specific harness configuration used during evaluation.
LLMs struggle to reliably apply skills, with performance varying significantly based on the agent harness used, challenging assumptions about their capability.
Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.
