Search papers, labs, and topics across Lattice.
This paper introduces PATS, a novel policy-centric training framework that enhances long-horizon reinforcement learning for LLM agents by treating skills as a dynamic training scaffold rather than static components. By converting rollout groups into evidence cards and employing task-specific evaluations, PATS provides concrete guidance that helps weak policies navigate complex tasks more effectively. The results show that PATS outperforms strong baselines by up to 18.6% in task completion and reduces prompt token usage by 32.1% across various benchmarks, demonstrating its efficiency and effectiveness in policy optimization.
Weak policies can achieve up to 18.6% better performance with PATS, a training framework that dynamically adapts guidance based on evolving policy needs.
In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, Pats, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. On ALFWorld and WebShop, Pats improves over strong baselines by up to 18.6%. Across seven search-augmented QA benchmarks, it remains competitive while using 32.1% fewer prompt tokens than the baseline.
