Search papers, labs, and topics across Lattice.
This paper introduces ContinualSkillBench, a dynamic evaluation framework designed to assess the ability of large language model (LLM) agents to evolve their skills through in-context continual learning across five domains with interconnected subtasks. The research reveals that while sequential execution generally enhances performance, the extent of improvement varies significantly across different models and domains, indicating that adaptation to prior context plays a crucial role. Notably, the study finds that less capable models tend to develop fragmented skill collections, highlighting the challenges in achieving robust and transferable skills through current in-context learning mechanisms.
In-context learning can match explicit skill maintenance in performance, but the real challenge lies in consolidating experience into effective, transferable skills.
Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.
