Search papers, labs, and topics across Lattice.
This paper introduces SENTINEL, a failure-driven reinforcement learning framework designed to enhance the training of tool-using language model agents by converting their rollout failures into targeted training tasks. By employing a Controller--Proposer--Solver loop, SENTINEL identifies recurring error patterns from failed trajectories and generates specific tasks aimed at addressing these weaknesses. The framework significantly improves performance on the Tau2-Bench Retail benchmark, increasing the Pass^1 score from 66.4 to 74.9 and outperforming traditional reinforcement learning on various synthetic tasks, highlighting the utility of leveraging model failures for effective training.
Transforming failures into focused training tasks boosts tool-using language model performance by over 8% on key benchmarks.
Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.
