Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Selective Teacher guided Policy Optimization (SeTPO) to enhance tool learning in small language models (SLMs) by combining selective teacher guidance with persistent demonstration supervision. SeTPO addresses the limitations of pure RL and direct supervision by incorporating reverse KL-derived token-level teacher feedback into policy advantages for exploration and forward KL supervision to preserve teacher behaviors. Theoretical analysis shows that SeTPO's bidirectional KL objective yields an optimal policy with at least the teacher's expected reward and a guaranteed fraction of the teacher's probability mass, empirically validated by superior performance on BFCL v4 and API Bank datasets across various SLM scales.
Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.
External tools can extend the capabilities of small language models (SLMs), enabling their deployment in resource constrained environments. However, their limited capabilities make it difficult for SLMs to discover effective tool use behaviors through reinforcement learning alone, while direct teacher supervision suffers from teacher student behavioral distribution mismatch and may propagate erroneous preferences. To address these challenges, we propose Selective Teacher guided Policy Optimization (SeTPO), which coordinates student exploration with selective teacher guidance and persistent demonstration supervision. Specifically, SeTPO incorporates reverse KL derived token level teacher feedback into policy advantages on low reward trajectories, while auxiliary forward KL supervision preserves teacher demonstrated behaviors throughout training. Under an idealized bidirectional KL objective, we theoretically establish that the optimal policy achieves expected reward no lower than the teacher's while retaining a guaranteed fraction of the teacher's probability mass. Extensive experiments on BFCL v4 and API Bank demonstrate that SeTPO achieves superior overall performance over supervised fine tuning, reinforcement learning, and distillation baselines across all three student scales.