Search papers, labs, and topics across Lattice.
This paper introduces SkillSmith, an augmented LLM that integrates parametric skill libraries with textual knowledge to enhance problem-solving capabilities. By treating model weights as a reasoning modality, SkillSmith employs prefix-tuning to synthesize new prefix weights based on rich textual data, enabling instruction-steered parametric synthesis. The results show significant performance improvements over traditional text-only and weight-space-only approaches, highlighting the benefits of a multimodal integration strategy.
SkillSmith achieves unprecedented performance gains by seamlessly merging textual knowledge with parametric skills, outperforming traditional methods that treat these domains separately.
Agentic systems driven by large language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures from past experiences and building parametric (weight-space) skill libraries for recurring sub-goals. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight-space merging. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over. We instantiate parametric learning via prefix-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill. We demonstrate that our approach significantly outperforms both text-only and weight-space-only baselines, unlocking performance gains that are out of reach for uni-modal (text-only or weight-only) adaptations.
