Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SkillScaffold addresses the inefficiency of skill-based agentic RL by introducing a policy-adaptive training scaffold that estimates skill utility via outcome-aware likelihood comparisons on existing trajectories, avoiding costly environment rollouts. This approach selectively injects skills only when beneficial, allowing the generator to co-evolve with the executor policy. Experiments across ALFWorld, WebShop, ScienceWorld, and $τ^2$-Bench show significant performance improvements over strong baselines, highlighting the effectiveness of utility-based selection and abstention.
LLM agents can learn complex tasks more efficiently by adaptively injecting skills only when their utility is confirmed through outcome-aware likelihood comparisons on existing data, bypassing expensive environment rollouts.
Agentic reinforcement learning (RL) enables large language models (LLMs) to learn from multi-turn interactions with verifiable outcomes. Textual skills can improve exploration by distilling prior interactions into reusable guidance. However, existing skill-based RL approaches often either assume skill injection is beneficial or rely on indirect feedback or rollout-intensive evaluation to assess its utility. Our pilot studies show that rollout-based utility estimation for skill credit assignment is costly and unstable under limited sampling, while skill benefits vary across candidates and depend on the current policy's competence on each query. To address these challenges, we introduce SkillScaffold, a policy-adaptive training scaffold that combines low-cost skill-utility estimation with selective skill injection. By reusing outcome-verified training trajectories, SkillScaffold generates candidate skills and estimates their utility through outcome-aware likelihood comparisons, without candidate-specific environment rollouts. The resulting utility guides whether and which skill to inject, retaining skill-free execution when no candidate is estimated to help. Joint training enables the Generator to produce skills tailored to the evolving Executor. Scaffolded RL encourages skill internalization for skill-free inference. Experiments on ALFWorld, WebShop, ScienceWorld, and $τ^2$-Bench demonstrate gains over strong agentic-RL and skill-based RL baselines across both short- and long-horizon tasks. Ablations and further analyses support the importance of utility-based selection and abstention. Our code is available at https://github.com/Vaidurya00/SkillScaffold.